Dean Wampler's Mind

Groovy, JRuby, vs. Jexl Performance, Using Java 5 and Java 6

Posted by Dean Wampler Mon, 01 Jan 2007 00:38:00 GMT

I just released v0.7.0 of Contract4J5, which now supports Groovy and JRuby as alternative scripting languages, in addition to Jexl, the original language used. Along the way, I collected some performance numbers for the three alternatives.

The following discussion is adapted from the README for the v0.7.0 release.

I ran the JUnit test suite with all three languages. The numbers below show the performance when using binary weaving and load-time weaving (LTW) and also the performance using the JVMs in JDK 5 and JDK 6.

These times are approximate “user times”, averaged over a few runs, measured on a ThinkPad T42 running Ubuntu Linux. There are slightly different build activities and I/O overhead involved in the LTW numbers, adding a few percentage points to the numbers. The tests do some different manipulations depending on which interpreter is used. Hence, the results are rough, at best. Most likely, the numbers reflect the relative amounts of overhead to load the interpreters and to set up the “environments” for evaluating the scripts, which happens frequently in the test suite. Since the scripts are always very small, evaluation is probably not a large percentage of the execution time. (However, this is speculation!) Your mileage may vary…

Note that the JDK 6 performance is significantly better. The code was compiled with the JDK 5 javac and executed using the JDK 6 JVM. Ironically, the performance was slightly slower when compiled with the JDK 6 javac. (This is not reflected in the numbers)

The Jexl test runs are roughly twice as fast as the Groovy and JRuby runs, probably because Jexl is a more “minimalist” environment, incurring less startup and execution overhead.

While not shown, I observed that the memory requirements for Groovy and JRuby are also higher, as you would expect. I didn’t actually measure memory usage, but I observed this because it was necessary to increase the maximum heap size for the LTW JUnit run when using Groovy or JRuby. Also, when using JRuby, I had to increase the heap size for the binary-weaving JUnit runs in Eclipse.

The LTW test runs take significantly longer, but the results are somewhat hard to interpret. Under JDK 5, the Jexl and Groovy runs are five to six times longer, while the JRuby runs are only about 2.5 times longer. Under JDK 6, the Jexl and Groovy runs are a little more than three times longer, while the JRuby run is under twice as long. It is not clear why JRuby LTW configurations perform so much better under both JDKs. However, it is clear that, for all the interpreters, the startup byte-code instrumentation required for LTW is much better under JDK 6.

In general, the longer LTW times occur because a weaving step happens at the beginning of each TestCase, as the class is loaded. The JDK 6 speed-up is a good argument for running this VM, at least for your tests!

LTW is the most convenient way to adopt Contract4J5 (true for other aspects, as well), but if the startup time is too long for your needs, then binary weaving of compiled code provides better performance with only a modest change in your build process.

Posted in AOP, Java and AspectJ, Contract4J | no comments

Contract4J5 v0.7.0 Is Now Available

Posted by Dean Wampler Mon, 01 Jan 2007 00:19:00 GMT

I just released v0.7.0 of Contract4J5. This release adds support for using Groovy or JRuby instead of Jexl as the scripting engine for the test scripts embedded in annotations. In fact, Groovy is now the default.

The advantage of Groovy or JRuby is that they provide a richer environment for more sophisticated testing. They are also better suited for planned, long-term enhancements of Contract4J5 to support user-configurable annotations and behaviors, which will take the tool beyond Design by Contract support.

The only major disadvantage over Jexl is lower performance. I’ll blog about those results next. You can also find details in the README.

I was pleased to find that all but a few of the existing Jexl-based unit tests worked without modification for Groovy. With a few hacks, I got them to work for JRuby as well. The differences for Groovy include slight differences in how Goovy vs. Jexl handle protected access (private, protected, etc.). Also, Groovy does not force you to provide a public accessor to evaluate tests on member fields, which Jexl required.

For JRuby, I found that almost all Ruby vs. Java differences could be handled if I did a few simple replacements of the most common Java idioms in scripts with the corresponding Ruby idioms:

• Replace equals(...) with eql?(...)
• Replace compareTo(...) with <=>(...)
• Replace null with nil

That is, Contract4J5, when using JRuby, will make these substitutions in your test expressions so they work, most of the time, even though they are written in Java syntax. This tends to work because the test expressions tend to be relatively simple and they often consist of comparisons to null or calls to String comparison methods.

Posted in AOP, Java and AspectJ, Contract4J | no comments