XMUltra

Recently, XMUltra was released on SourceForge.net. This had been the feed processing engine for Knight Ridder Digital, the online division of the once-mighty Knight Ridder newspaper company. (No, we don't make talking cars).

Although originally designed for news feeds, it was expanded for use in a number of other areas. It has acquired a fairly large number of utilities, including tools for transferring files, running scripts, compressing files, etc.

In short, it has become to feed processing what Ant is to build processes.

XMUltra is available at http://xmultra.sourceforge.net now. If you are involved in Java-based data feed processing, you should check it out. However, I thought I'd talk a little about the history of the product today.

In The Beginning...

Before XMUltra, KRD processed news feeds with a collection of Perl scripts, shell scripts, C code, and (for some odd reason) the print spooler. These scripts did their job, but they were becoming increasingly difficult to maintain. We were becoming a Java shop, and it was decided that we should have a Java-based feed processing engine.

Total rewrites tend to fail. Usually, you are better off refactoring your existing code base. In this case, however, XMUltra was a complete success. It was hailed as a triumph, not only by Knight Ridder, but even by the Newspaper Association of America.

The Architect

Wayne Weber was the perfect person to head this charge. He had a clear long-term vision for the product, a passion for technology, and perhaps most critically, an amazing mind for details.

I've thought about the best way to describe Wayne, but I think the best way is to show an excerpt of his code:

try {
   // Using reflection to avoid compiler warnings for "stop" (heh, heh).
   Class classObj = Thread.class;
   Method method = classObj.getMethod("stop", null);
   method.invoke(processorThread, null);
} catch (Exception e) {
...

My first thought was to nominate this for The Daily WTF. Except, it is not really a WTF. It can't be when there is a clear comment about what exactly he was doing. There was a section I left out that explained the reasoning for using the deprecated method.

More importantly, it speaks to something deeper. Wayne went through some elaborate hoops just to avoid any compile warnings. Despite the large number of classes, there were absolutely zero warnings and no methods or classes without JavaDoc comments. No one else was allowed to get away with anything less than perfection.

Expansion

XMUltra was primarily designed for news feeds. Among other things, it featured a whole plugin architecture for news feeds. However, it also had a variety of extra general-purpose utilities, and gradually we began to use XMUltra wherever we could.

This is really where I first became involved. I extended XMUltra for processing classified ads. Writing new processors was somewhat painful at the time, but it was well worth it. Over time, through the efforts of the KRD team, XMUltra gathered a richer and richer collection of utilities.

Mothballed

There was one problem with XMUltra. It aged. Codebases become out of date if they are not maintained. And while we spent our efforts on adding new tools, we did not have the luxury of updating the core architecture.

It shows in a few areas. Built before log4j was well-established, XMUltra features its own custom logging framework. There was a well established bulk-test, but no unit-tests. When Java 5 was released, XMUltra went from warning-free to having several hundred warnings.

XMUltra seemed doomed after McClatchy merged with Knight Ridder. McClatchy Interactive was a Perl and Ruby shop, and Java applications have gradually been phased out. XMUltra is still in fairly heavy production use, but it is clear that it does not have a future within the organization.

Wayne himself has left McClatchy Interactive now. Most of the rest of the core XMUltra team was gone already. I am the last.

Resurrection

McClatchy Interactive agreed to let me open source XMUltra. Before he left, Wayne and I spent a couple of days cleaning up the build process, stripping out MI/KRD specific components, adding some examples, and updating the documentation. It is not in perfect condition, but it is in a form ready for public consumption.

Check it out! I hope you find it as useful as I have.