Monitoring Java Processes with JMX

Step 0. If hostname -i does not report the public IP address of the system, then edit /etc/hosts and set the public IP there. For example, on cn-dev, hostname -i reported 127.0.1.1. The hosts file was updated with the correct value:

127.0.0.1     localhost
#127.0.1.1    cn-dev.dataone.org      cn-dev
128.111.220.50        cn-dev.dataone.org      cn-dev

...

See: http://docs.oracle.com/javase/1.5.0/docs/guide/management/faq.html#linux1

Watching Hazelcast

In this example, d1-processing is enabled for JMX monitoring.

Create the file /etc/dataone/process/jmx.passwd with contents:

monitorRole {PASSWORD}

and the file /etc/dataone/process/jmx.access with contents:

monitorRole readonly

Change owners of these to user tomcat6 and make them readable only by that user (has to be same user as process that will be launching the JMX service):

sudo chown tomcat6:tomcat6 /etc/dataone/process/jmx.*
sudo chmod 600 /etc/dataone/process/jmx.*

Shutdown d1-processing:

sudo /etc/init.d/d1-processing stop

now startup d1-processing with the JMX startup flags:

sudo env JAVA_OPTS="-Djava.awt.headless=true -Xmx4096M -Xms1024M \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=8010 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.password.file=/etc/dataone/process/jmx.passwd \
  -Dcom.sun.management.jmxremote.access.file=/etc/dataone/process/jmx.access \
  -Djava.rmi.server.hostname=128.111.220.50 \
  -Dhazelcast.jmx=true" \
  /etc/init.d/d1-processing start

Temporarily disable the firewall. This is necessary because even though the JMX service will listen on the specified port, the RMI service, which the JMX client will be directed to by the JMX service, will be listening on a random port:

sudo ufw disable

Open jconsole on your desktop, and select “Remote process”, entering in:

hostname:port

and the username “monitorRole” and the password specified in /etc/dataone/process/jmx/passwd.

After a couple of seconds the JMX client should be connected and start collecting statistics.

Remember to restart the firewall when you’re done:

sudo ufw enable

Watching Tomcat

There’s an issue with Java security, probably need permission to access the password and access files, but as an interim measure, disable JAVA_SECURITY in /etc/init.d/tomcat6, stop tomcat6, and restart with the following parameters to enable JMX monitoring of tomcat:

sudo env JAVA_OPTS="-Djava.awt.headless=true -Xmx2048M -Xms1024M \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=8020 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.password.file=/etc/dataone/process/jmx.passwd \
  -Dcom.sun.management.jmxremote.access.file=/etc/dataone/process/jmx.access \
  -Djava.rmi.server.hostname=128.111.220.50 \
  -Dhazelcast.jmx=true" \
  /etc/init.d/tomcat6 start

sudo env JAVA_OPTS="-Djava.awt.headless=true -Xmx2048M -Xms1024M \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=8020 \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.password.file=/etc/dataone/monitor/jmx.passwd \
  -Dcom.sun.management.jmxremote.access.file=/etc/dataone/monitor/jmx.access \
  -Djava.rmi.server.hostname=129.24.0.109 \
  -Dhazelcast.jmx=true" \
  /etc/init.d/tomcat6 start

Check jmx tool:

Usage: check_jmx -U url -O object_name -A attribute [-K compound_key] [-I attribute_info] [-J attribute_info_key] -w warn_limit -c crit_limit [-v[vvv]] [-help]

, where options are:

-help  Prints this page

-U   JMX URL, for example: "service:jmx:rmi:///jndi/rmi://localhost:1616/jmxrmi"

-O   Object name to be checked, for example, "java.lang:type=Memory"

-A   Attribute of the object to be checked, for example, "NonHeapMemoryUsage"

-K   Attribute key for -A attribute compound data, for example, "used" (optional)

-I   Attribute of the object containing information for text output (optional)

-J   Attribute key for -I attribute compound data, for example, "used" (optional)

-v[vvv] verbatim level controlled as a number of v (optional)

-w   warning integer value

-c   critical integer value

Note that if warning level > critical, system checks object attribute value to be LESS THAN OR EQUAL warning, critical
If warning level < critical, system checks object attribute value to be MORE THAN OR EQUAL warning, critical



./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8020/jmxrmi \
  -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage \
  -J used -vvvv -w 4248302272 -c 5498760192

./check_jmx -U service:jmx:rmi:///jndi/rmi://localhost:8020/jmxrmi \
-O java.lang:type=Memory -A LoadedClassCount -K used -I HeapMemoryUsage -J used -vvvv -w 4248302272 -c 5498760192

Listing JMX Beans

Get a JMX console tool. The one used in the examples here is jmxterm available from: http://wiki.cyclopsgroup.org/jmxterm

Fire up jmxterm with something like java -jar jmxterm.jar, then connect to the target using the open command:

java -jar jmxterm.jar
$> open 127.0.0.1:8020
#Connection to 127.0.0.1:8020 is opened

Get a list of domains:

$>domains
#following domains are available
Catalina
JMImplementation
Users
com.sun.management
java.lang
java.util.logging
solr/

Select a domain, in this case Catalina and see what beans it offers:

$>domain Catalina
#domain is set to Catalina
$>beans
#domain = Catalina:
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/,j2eeType=Servlet,name=default
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/,j2eeType=Servlet,name=jsp
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/,name=jsp,type=JspMonitor
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Filter,name=SolrRequestFilter
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Servlet,name=Logging
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Servlet,name=SolrServer
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Servlet,name=SolrUpdate
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Servlet,name=default
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Servlet,name=jsp
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,j2eeType=Servlet,name=ping
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,name=jsp,type=JspMonitor
Catalina:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/solr,name=ping,type=JspMonitor
Catalina:J2EEApplication=none,J2EEServer=none,j2eeType=WebModule,name=//localhost/
Catalina:J2EEApplication=none,J2EEServer=none,j2eeType=WebModule,name=//localhost/solr
Catalina:class=org.apache.catalina.UserDatabase,name="UserDatabase",resourcetype=Global,type=Resource
Catalina:host=localhost,name=ErrorReportValve,type=Valve
Catalina:host=localhost,name=StandardContextValve,path=/,type=Valve
Catalina:host=localhost,name=StandardContextValve,path=/solr,type=Valve
Catalina:host=localhost,name=StandardHostValve,type=Valve
Catalina:host=localhost,name=solr/home,path=/solr,resourcetype=Context,type=Environment
Catalina:host=localhost,path=/,resourcetype=Context,type=NamingResources
Catalina:host=localhost,path=/,type=Cache
Catalina:host=localhost,path=/,type=Loader
Catalina:host=localhost,path=/,type=Manager
Catalina:host=localhost,path=/,type=WebappClassLoader
Catalina:host=localhost,path=/solr,resourcetype=Context,type=NamingResources
Catalina:host=localhost,path=/solr,type=Cache
Catalina:host=localhost,path=/solr,type=Loader
Catalina:host=localhost,path=/solr,type=Manager
Catalina:host=localhost,path=/solr,type=WebappClassLoader
Catalina:host=localhost,type=Deployer
Catalina:host=localhost,type=Host
Catalina:name=StandardEngineValve,type=Valve
Catalina:name=common,type=ServerClassLoader
Catalina:name=http-8080,type=GlobalRequestProcessor
Catalina:name=http-8080,type=ThreadPool
Catalina:name=server,type=ServerClassLoader
Catalina:name=shared,type=ServerClassLoader
Catalina:port=8080,type=Connector
Catalina:port=8080,type=Mapper
Catalina:port=8080,type=ProtocolHandler
Catalina:realmPath=/realm0,type=Realm
Catalina:resourcetype=Global,type=NamingResources
Catalina:serviceName=Catalina,type=Service
Catalina:type=Engine
Catalina:type=MBeanFactory
Catalina:type=Server
Catalina:type=StringCache

The Host bean looks interesting:

$>bean Catalina:host=localhost,type=Host
#bean is set to Catalina:host=localhost,type=Host
$>info
#mbean = Catalina:host=localhost,type=Host
#class name = org.apache.tomcat.util.modeler.BaseModelMBean
# attributes
  %0   - aliases ([Ljava.lang.String;, rw)
  %1   - appBase (java.lang.String, rw)
  %2   - autoDeploy (boolean, rw)
  %3   - children ([Ljavax.management.ObjectName;, rw)
  %4   - configClass (java.lang.String, rw)
  %5   - deployOnStartup (boolean, rw)
  %6   - deployXML (boolean, rw)
  %7   - managedResource (java.lang.Object, rw)
  %8   - modelerType (java.lang.String, r)
  %9   - name (java.lang.String, rw)
  %10  - realm (org.apache.catalina.Realm, rw)
  %11  - unpackWARs (boolean, rw)
  %12  - valveNames ([Ljava.lang.String;, rw)
  %13  - valveObjectNames ([Ljavax.management.ObjectName;, rw)
  %14  - xmlNamespaceAware (boolean, rw)
  %15  - xmlValidation (boolean, rw)
# operations
  %0   - void addAlias(java.lang.String alias)
  %1   - void addChild(org.apache.catalina.Container child)
  %2   - void destroy()
  %3   - [Ljava.lang.String; findAliases()
  %4   - void init()
  %5   - void removeAlias(java.lang.String alias)
  %6   - void start()
  %7   - void stop()
#there's no notifications

Now let’s get a couple attribute values:

$>get appBase
#mbean = Catalina:host=localhost,type=Host:
appBase = webapps;

$>get children
#mbean = Catalina:host=localhost,type=Host:
children = [ Catalina:j2eeType=WebModule,name=//localhost/,J2EEApplication=none,J2EEServer=none, Catalina:j2eeType=WebModule,name=//localhost/solr,J2EEApplication=none,J2EEServer=none ];

This material is based upon work supported by the National Science Foundation under Grant Numbers 083094 and 1430508.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.