Data Aggregation via JMX and the Grid

Following my post on JMX and the Grid which got picked up by The Server Side and by Nati Shalom's blog here I thought I'd add some more brief thoughts on another complimentary JMX pattern we've used in conjunction with grid applications.

The original post talks about collating client-side access to a distributed population of JMX MBeans that comprise the application. In essence the technique described in that post to use a JavaSpace (or other rendezvous technology) to act as a point of registration and lookup. This gives the client-side access to (say) a list of MBeans for each instance of a given type component wherever its running in the grid and the ability for the agent to communicate with any MBean to get/set attributes, invoke management operations or hear notification events.

The client-side (or "agent") of JMX is by nature pretty dumb. Generally the agent uses metadata info about the MBean to generate a UI on the fly. Although it's possible to write custom JMX agents for your application (and we do that), to make sure your management MBeans will work with any JMX agent you really have to design to the lowest common denominator agent.

So let's consider the use-case where our MBeans are collecting stats about (say) our application's performance: average task execution time, latency etc. Stats can be produced for each individual component and made available via the MBean, but we also want to be able to see an aggregated view statistics for the application as a whole.

Aggregation

To deal with the dumb JMX agent we really need to collate and aggregate the data server-side. I'm not going to dwell too much on the approach to this, other than to say aggregation might be done in one of three ways:

1. Writing an server-side component that collects stats from individual MBeans and aggregates. In this case, using the approach outlined in my previous JMX piece might be handy
2. Tapping into the underlying components using some application-specific API and aggregating from there
3. Having the components publishing their stats into a JavaSpace and having an aggregating component attached to the space to perform the aggregation.

Focussing on the last of these approaches for a moment, using the space as a rendezvous point for collation and aggregation has some merits: publication of stats as POJOs to the space is easy and listening to those publications to trigger aggregation is also simple to implement.

Publication to JMX

Regardless of the approach to aggregation, we also need a technique for making the aggregated stats available to dumb JMX agent. The aggregating component needs to expose an MBean to provide access to the aggregated data values. In a simple application these can be held as in-memory values within the aggregating component. However, to deal with large data volumes and to provide fault-tolerance we prefer the following approach:

1. Aggregating components write the results back to the JavaSpace
2. A stateless component provides an MBean that acts as a facade to the aggregated data, which is actually fetched on demand from the space

Using the GigaSpaces product we can rely on the space itself to manage live reliable backup of our aggregated data and the Service Grid to host and maintain our stateless aggregated MBean facade.

Summary

Although in our simple aggregating stats use-case we might not care about dropping data or fault-tolerance, there are many real-world examples where we would care far more about these issues. The bare-bones architecture of using the space as both a rendezvous point and a safe holding repository, with access via stateless service components applies well.

One of the reasons I'm a fan of GigaSpaces and space-based architectures is that a number of architectural choices that are traditionally hard-wired: transactional/non-transactional, sync or async replication can be changed through configuration only. This enables common design patterns (and therefore components) to be applied to a wide range of application problems, by enabling the data integrity/performance equation to be tweeked at a late stage of application assembly.

I know this last paragraph is a bit of a leap from the initial topic, but I'll return to this theme in later postings which discuss other use-cases where data integrity and fault-tolerance are a significant issue, in an attempt to make it stand up.

OSGI, GigaSpaces and Buddy Classloading

I've been doing a reasonable amount of GUI development using Eclipse Rich Client Platform recently. This is a great framework and getting better. In the Eclipse 3.1 dark days I got a bit burned trying to build applications, largely because the development support for RCP within Eclipse was itself a bit flakey. There's nothing more galling than spending half a day trying to diagnose a problem only to find that restarting Eclipse clean sorts it all out. But I digress...

Eclipse RCP and Plugin Architecture

I won't wax too lyrical about the benefits of the Eclipse plugin model. If you are a Java developer you probably already understand the great value of being able to drop in plugins from around the web and rely on Eclipse to run with simultaneous multiple versions of the same jar. At my company, PSJ, we've been developing a number of operations console GUIs that interact with application services and the GigaSpaces implementation of JavaSpaces. The plugin model provides a great basis for us to build small UI plugins that interact with different service components we've developed over the years. Using Spring we're then able to wire together our re-usable plugins and UI pieces with custom components written in the context of an engagement into a customer-specific GUI.

Eclipse's ability to manage multiple versions of classes concurrently is ultimately down to the multiple classloader OSGI mechanism that underpins the plugin architecture. So far so triffic, however if you've ever tried to use plugins with technologies like Spring, JavaSpaces even Log4j, you'll have encountered ClassNotFoundException problems galore. Reading around the web when I first hit this a couple of years ago the recommendations were to hack in and manipulate the classloaders in application code.

ClassLoader loader = Thread.currentThread().getContextClassLoader();
ClassLoader thisClassesLoader = this.getClass().getClassLoader();
Thread.currentThread().setContextClassLoader(thisClassesLoader);
// do your stuff here
Thread.currentThread().setContextClassLoader(loader);

Not only was this pretty horrible, but it's very difficult to make bombproof as you've got to track down all points where the issue can occur and problematic when using technogies like GigaSpaces that use internal thread pools even for synchronous calls and deliver events to you on their own threads.

In revisiting this topic again recently in some work for a client, I'm grateful to my colleague John Nichol who has shown me the one true way: buddy classloaders. Like many things with Eclipse RCP the documentation of this is vanishingly thin and largely what I call "non-doc" - you know the kind of thing:

"To press button B, click the B button"

Arggh! Anyway after trawling around the web and single-step debugging within the OSGI class resolution code, I have mined the following nuggets of true knowledge.

Why is ClassNotFound in the First Place?

If you are using technologies like Spring or GigaSpaces, they both need access to the classes in your application code - in one case to instantiate instances and wire them together, and in the other to store instances in a sharable in-memory location on the network. If you put the Spring and GigaSpaces jar files inside your plugin you can't easily share instances of objects across plugins. The solution to this is to create independent plugins for Spring, GigaSpaces, Log4j etc and then build dependencies between your higher level plugins and these lower level ones. So far so good, but this is where you can hit ClassNotFoundException. Let's say Spring needed to instantiate an instance of your application class Foo. Well Foo isn't in the Spring plugin and you're screwed. You could also wodge all the application code, Spring, Gigaspaces jars together into one big plugin, but then really you've lost the advantage of component separation you were trying achieve in the first place. You'd also be back to manually manipulating the thread context classloaders. So, you're screwed, right...

Buddy Classloading

Fortunately to get you out of this jam Eclipse has a mechanism called buddy classloading. This lets you add directives to the plugin manifests to selectively delegate classloading to other friendly plugins. Add the following line:

Eclipse-BuddyPolicy: registered

To the lower-level Spring, GigaSpaces plugin manifests. This tells those plugins that they can delegate class-loading to any plugin that registers with them. You also need to add directives to your application-level plugins to perform registration with lines in their manifests like:

Eclipse-RegisterBuddy: org.springframework,com.gigaspaces, org.apache.log4j,org.apache.commons

Going back to our Foo class example earlier, when the Spring plugin tries to instantiate a Foo instance it will fail to resolve the class from its own plugin and will then attempt to resolve from any buddies that it knows about. Foo therefore gets resolved from the application plugin that is registered with the Spring plugin. This happens no matter what thread is attempting to resolve the class and doesn't suffer from the holes that context classloader manipulation suffered from.

Direction of Buddy Registration

One point to make clear here (coz it tripped me up when I was trying to get my feeble brain around it all) is the direction of registration. I had originally thought from reading the non-doc that registration was spring, gigapsaces plugins registering with the application plugin. Actually it's the reverse. Application plugins register with spring, gigaspaces because they want their classes to be accessible to those generic technologies for instantiation purposes. The confusion arises because only the application plugin can specify which generic plugins it wants to register with and the manifest entry Eclipse-RegisterBuddy implies (to me at least) that the list that follows the entry is registering with the current plugin.