JMX For Grid-Based Applications

This blog talks about a (fairly) simple technique that solves the problem of how you get a unified view of JMX Management Beans in a distributed application. This is proving useful in a number of application projects we are doing at PSJ where the application MBeans, for example worker beans are distributed around the network when deployed in the GigaSpaces Service Grid. The technique describes how a new protocol can be added to enable JMX agents and JMX servers to find each other in a network deployment. The protocol uses the GigaSpace as the rendezvous point, but the approach can be easily adapted to work with other network registration and rendezvous technologies.

A Brief History of JMX

JMX has been around for a long time as one of the core APIs in J2EE and recent versions of Java have seen it incorporated into the JVM to provide memory and other basic stats. The basics of JMX are pretty simple: you instrument your applications by providing one or more management beans MBeans. MBeans provide read-only and read-write attributes, and operations that provide information and enable the application's management characteristics to be controlled. MBeans are published to the world by registering them with an MBeanServer. MBeans are interacted with by a separate agent, which finds MBeans in the MBeanServer and provides a UI to interact with them. Often this UI takes the form of a generic user interface based on properties provided by the MBean, or as specific to the application as required.

The original JMX specs were charmingly vague about where you might find an MBeanServer to register with or how you'd connect to one from the agent side. In more recent times, the JVM provides a default MBeanServer inside the JVM. As well as providing management capability for the JVM this MBeanServer can act as a repository for application MBeans. The jconsole provided with Java 1.6 provides a decent, if primitive, GUI for you to see the MBeans registered with the JVM's MBeanServer and interact with them.

JMX JSR-160 Connectors

JMX also now provides connectors that facilitate remote connection between agent and server, and come in two flavours: server-side and client-side. The server-side connector enables you to set up a connection channel to talk to the MBeanServer remotely, for example by exporting an RMI stub. The client-side connector provides the means of binding to the server-side connector and establishing a connection, for example via JNDI lookup of the server-side connector's RMI stub.

If you use Spring, you can simply declare connectors in Spring configuration, as follows:

<bean id="serverConnector"
class="org.springframework.jmx.support.ConnectorServerFactoryBean">
<property name="objectName" value="system:name=spaceconnector"/>
<property name="serviceUrl”
value="service:jmx:rmi://localhost/jndi/rmi://localhost:1099/test"/>
<property name="environment">
<map>
 <entry key="jmx.remote.jndi.rebind" value="true" />
</map>
</property>
</bean>

The client-side connector is declared as:

<bean id="clientConnector"
class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean">
<property name="serviceUrl"
value="service:jmx:rmi://localhost:1099"/>
</bean>

JMX Issues Specific to the Grid

For monolithic applications that run in a single JVM this architecture works fine, but when applied to distributed applications running in the grid we have two additional problems:

1. How does the client-side find all the MBeans that comprise the application?
   
2. How can it interact with the distributed parts of the application from a single client?
   

This has become a real-world problem for PSJ in implementing grid-deployed applications within the GigaSpaces Service Grid and other grid fabrics. Here's the nub of the solution we came up with to solve these problems. In essence the solution has two parts:

• Provide a means of binding the MBeanServers from individual JVMs into a community that represents the application on the grid.
  
• Collate the MBeanServers together to provide a single client-side federated connector.

Communities of MBeanServers

Fortunately JMX provides a very open means of extending the protocols that are supported in bringing JMX server and client connections together. By providing an additional protocol we can control the server-side connector's registration process and the client-side connector's binding mechanism. There are various options to achieve our ends here including using naming hierarchies in JNDI, JINI lookup groups, but the one we settled on uses a JavaSpace to act as a rendezvous point for servers and clients. The rationale here is partially simply expedience: the applications to which we've applied this pattern already use a GigaSpace to share state and so in some senses the community is bound by the fact that all components comprising the application point at the same GigaSpace instance. The second rationale is that adding space-based registration and client-side lookup using the GigaSpace is very quick and easy to write, using simple POJOs to represent the registration.

Adding a Space-Based Server-Side JMX Connector

The standard RMI-based JMX URL can be changed by replacing the service.jmx.rmi part with service:jmx:space to indicate that we want to use a space protocol. The JMX spec lets us add handler classes to deal with the "space" protocol referenced in the URL on the server and client sides. On the server-side we need to provide a specific ServerProvider class implementing JMXConnectorServerProvider. The ServerProvider actually just piggy-backs on the existing RMI one and in addition to RMI stub registration places an entry in the JavaSpace with the RMI connection URL needed by the client. To finish off, using the Spring approach we simply need to declare the server connector to use the new protocol:

<bean id="serverConnector"
class="org.springframework.jmx.support.ConnectorServerFactoryBean">
<property name="objectName" value="system:name=spaceconnector"/>
<property name="serviceUrl"
value="service:jmx:space://localhost/jndi/rmi://localhost:1099/test"/>
<property name="environment">
<map>
  <entry key="jmx.remote.jndi.rebind" value="true" />
  <entry key="jmx.remote.protocol.provider.pkgs"
    value="com.psjsolutions.sflib.jmx.protocols"/>
  <entry key="space"><ref bean="space"/></entry>
</map>
</property>
</bean>

Notice that we've had to reference our protocol support package in the environment map for the connector factory. We can also place protocol-specific properties in the environment - in this case the space we want to use to hold the entries.

The Client-Side Connector

I said earlier that there are two problems to overcome in applying JMX to the grid: one being rendezvous/binding and the other being obtaining a collated view of all the MBeans out there. Both these issues are addressed by the client-side connector. First off we need a client-side connector that can look in the space to find all the MBeanServer connection details for the networked community. This is pretty easy. In symmetry with the server-side provider all we need to do is to write a ClientProvider that understands the space protocol and provides the agent with a client connector. In Spring this looks like:

<bean id="clientConnector"
class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean">
<property name="serviceUrl"
value="service:jmx:space:///jini://*/*/${javaspace.name}"/>
</bean>

All we've done here is to replace the RMI-based URL with one that specifies the space protocol and provides the URL of the space. The ClientProvider parses the service URL extracting the space URL and using it to bind to the space and extract all the server connector details. This brings us to the final part, which is to provide a collated "virtual" server connection that sits between the client-side user code and the set of MBeanServers.

Federating the Client-Side Server Connection

As far as the client-side code is concerned the connector it gets is an object that implements javax.management.MBeanServerConnection. This API is actually pretty straight-forward, enabling MBeans to be found by naming and query patterns and using the found MBeanName handles to get/set attributes and invoke operations. In Java 7 there may well be a formally supported means of cascading or handing on these requests, but as of the time of writing there's no capability out of the box. We therefore implemented a FederatedMBeanServerConnection class that picks up a number of MBeanServer connections from the space, connects to them and then delegates operations to the set of servers, effectively acting as a multiplexer.

In Summary

As we like to carry around these common solutions to common problems from job-to-job, we've added the capabilities described here to our foundation libraries that we often use to implement client engagements. By seperating the federation/multiplexing capability from the space protocol we can use this approach in a number of different distributed architectures, and will probably add protocols as the need arises. The beauty of the approach is that neither the client or server side code that use the connectors know what's being done under the covers. It's all abstracted into protocol URLs and therefore simple configuration changes.

Are there any gotchas? The one we've hit so far is possible ambiguity in naming MBeans. In many senses the set of MBeanServers in the grid can be thought of as a single virtual networked MBeanServer. However, whilst you can't register the same named MBean twice with a single MBeanServer, there's nothing to stop you registering the same named MBean with different MBeanServers. In fact in a grid-style environment where a given application unit is deployed as many replicated instances it's quite likely that you will hit this issue. Why is this important:? Well inability to enforce unique names can lead to ambiguity when we use the virtual MBeanServer as multiple MBeans with the same name can be found in the virtual server. We can work around this to a large extent by collating MBean queries and tagging the owning server in the MBeanName handed back through the client connector. This disambiguates things in most of the use cases including attribute get/set and operation invocation. However if the client code asks for a specific MBean by name then all we can do is return the first one we encounter. This is not ideal, but in practice the way to solve this is to use naming strategies when registering MBeans server-side. Spring has the idea of a pluggable NamingStrategy for MBeans that are auto-created by Spring. Using the instance-based NamingStragey resolves this problem for Spring Mbeans and, if it is a real issue, this approach could also be adopted by application code that is using explicit creation/registration of MBeans.