How to monitor Java/Tomcat using check_jmx through firewall/Amazon VPC – Nagios/Icinga

After lot of research over search engine,I figured out the JMX monitoring over firewall/ Amazon VPC. I would like to describe here all bunch of informations which i learnt through blogs and forums.I hope, it will help some one who looking for the same kind of solution and I belive, This post can reduce his/her time and effort rather thn searching with lot of effort.

After completing setup of Nagios/Icinga, We can use check_jmx plugin to monitor Tomcat/java process, Thread Count, Garbage Collector and Heap Memory Usage.

Make sure your rmiregistry port is open and listening (here I used port 9696 for rmiregistry). if you are using VPC, enable the port access to the monitoring system by using security groups option at AWS console.

# nmap -p 9696 localhost

Starting Nmap 5.51 ( http://nmap.org ) at 2014-04-17 11:10 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000087s latency).
PORT     STATE SERVICE
9696/tcp open  rmiregistry

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

I always keep iptables enable for system, since it will proctect your system against both malicious users and software such as viruses/worms.

Add rmiregistry in iptables and /etc/services as well.

# cat /etc/services | grep rmiregistry
rmiregistry     9696/tcp                # RMI Registry

# iptables -L
target     prot opt source               destination
ACCEPT     tcp  —  anywhere             anywhere            state NEW tcp dpt:rmiregistry

Make sure your iptables added and enabled with chkconfig.

# chkconfig iptables on

Disable selinux,

# getenforce
Disabled

On client machine (Tomcat/java machine) edit your nrpe and add the commands as shown below, so that we can just execute the nrpe command from monitoring system over the particular port which you have given access to monitoring system through vpc/firewall.

# vi /etc/nagios/nrpe.cfg

#### JMX Monitoring ####

command[check_jmx_threadcount]=/usr/lib64/nagios/plugins/check_jmx -U service:jmx:rmi:///jndi/rmi://”hostname”:9696/jmxrmi -O java.lang:type=Threading -A ThreadCount -K “” Total

command[check_jmx_heap]=/usr/lib64/nagios/plugins/check_jmx -U service:jmx:rmi:///jndi/rmi://”hostname”:9696/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 102400 -c 81290

command[check_jmx_garbagecollector]=/usr/lib64/nagios/plugins/check_jmx -U service:jmx:rmi:///jndi/rmi://”hostname”:9696/jmxrmi -O “java.lang:type=GarbageCollector,name=Copy” -A LastGcInfo -K duration -w 3500 -c 4000 -u ms

Here you go to add check_jmx commands on your monitoring system,which you already defined with the client machine.

edit commands.cfg and add following to monitor heap,thread-count and garbage collector.

# ‘check_jmx’ command definition

define command{
command_name    check_jmx_heap
command_line    $USER1$/check_nrpe -H $HOSTADDRESS$  -c check_jmx_heap
}

define command{
command_name    check_jmx_current_threadcount
command_line    $USER1$/check_nrpe -H $HOSTADDRESS$  -c check_jmx_threadcount
}

define command{
command_name    check_jmx_garbagecollector
command_line    $USER1$/check_nrpe -H $HOSTADDRESS$  -c check_jmx_garbagecollector
}

Additional Note  : –

If you have space in arguments , normally check_jmx will not handle the request.
For example “‘java.lang:type=GarbageCollector,name=PS Scavenge'”  so you need to edit your check_jmx and update double quotes to $@ fix this.

Change in check_jmx (last line) From java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery $@   to  java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery “$@”

Updated check_jmx shown below,

# cat /usr/lib64/nagios/plugins/check_jmx
#!/bin/sh
#
# Nagios plugin to monitor Java JMX (http://java.sun.com/jmx)attributes.
#
RDIR=`dirname $0`
java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery “$@”

Result :-