How to monitor Java/Tomcat using check_jmx through firewall/Amazon VPC – Nagios/Icinga
After lot of research over search engine,I figured out the JMX monitoring over firewall/ Amazon VPC. I would like to describe here all bunch of informations which i learnt through blogs and forums.I hope, it will help some one who looking for the same kind of solution and I belive, This post can reduce his/her time and effort rather thn searching with lot of effort.
After completing setup of Nagios/Icinga, We can use check_jmx plugin to monitor Tomcat/java process, Thread Count, Garbage Collector and Heap Memory Usage.
Make sure your rmiregistry port is open and listening (here I used port 9696 for rmiregistry). if you are using VPC, enable the port access to the monitoring system by using security groups option at AWS console.
# nmap -p 9696 localhost
Starting Nmap 5.51 ( http://nmap.org ) at 2014-04-17 11:10 CEST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000087s latency).
PORT STATE SERVICE
9696/tcp open rmiregistryNmap done: 1 IP address (1 host up) scanned in 0.14 seconds
I always keep iptables enable for system, since it will proctect your system against both malicious users and software such as viruses/worms.
Add rmiregistry in iptables and /etc/services as well.
# cat /etc/services | grep rmiregistry
rmiregistry 9696/tcp # RMI Registry# iptables -L
target prot opt source destination
ACCEPT tcp — anywhere anywhere state NEW tcp dpt:rmiregistry
Make sure your iptables added and enabled with chkconfig.
# chkconfig iptables on
Disable selinux,
# getenforce
Disabled
On client machine (Tomcat/java machine) edit your nrpe and add the commands as shown below, so that we can just execute the nrpe command from monitoring system over the particular port which you have given access to monitoring system through vpc/firewall.
# vi /etc/nagios/nrpe.cfg
#### JMX Monitoring ####
command[check_jmx_threadcount]=/usr/lib64/nagios/plugins/check_jmx -U service:jmx:rmi:///jndi/rmi://”hostname”:9696/jmxrmi -O java.lang:type=Threading -A ThreadCount -K “” Total
command[check_jmx_heap]=/usr/lib64/nagios/plugins/check_jmx -U service:jmx:rmi:///jndi/rmi://”hostname”:9696/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 102400 -c 81290
command[check_jmx_garbagecollector]=/usr/lib64/nagios/plugins/check_jmx -U service:jmx:rmi:///jndi/rmi://”hostname”:9696/jmxrmi -O “java.lang:type=GarbageCollector,name=Copy” -A LastGcInfo -K duration -w 3500 -c 4000 -u ms
Here you go to add check_jmx commands on your monitoring system,which you already defined with the client machine.
edit commands.cfg and add following to monitor heap,thread-count and garbage collector.
# ‘check_jmx’ command definition
define command{
command_name check_jmx_heap
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_jmx_heap
}define command{
command_name check_jmx_current_threadcount
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_jmx_threadcount
}define command{
command_name check_jmx_garbagecollector
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_jmx_garbagecollector
}
Additional Note : –
If you have space in arguments , normally check_jmx will not handle the request.
For example “‘java.lang:type=GarbageCollector,name=PS Scavenge'” so you need to edit your check_jmx and update double quotes to $@ fix this.
Change in check_jmx (last line) From java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery $@ to java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery “$@”
Updated check_jmx shown below,
# cat /usr/lib64/nagios/plugins/check_jmx
#!/bin/sh
#
# Nagios plugin to monitor Java JMX (http://java.sun.com/jmx)attributes.
#
RDIR=`dirname $0`
java -cp $RDIR/jmxquery.jar org.nagios.JMXQuery “$@”
Result :-