A Tale of Four JVMs and One App Server

Starting point

A user has recently reported in the JBoss issue tracker that JBoss AS 5.0.0 GA fails to start on the IBM JRE. One may say: So what? Didn’t you hear that the IBM JRE “fits better” for another app server?

Independently, our QA team has encountered the same problem when testing JBoss 5.0 on a new version of Excelsior JET JVM. A quick check confirmed that the problem manifests itself on Oracle JRockit as well.

At the same time, on the Sun JRE everything worked like a charm…

Well, I respect the Sun reference implementation but something suggested to me that the other three JVMs, which all support Java 6 and passed the Sun JCK, cannot have exactly the same bug.

Further investigation has shown that the issue was in the Java code that worked on the Sun JRE by a lucky coincidence.

Just follow the specification

The root cause of the problem is that JBoss 5.0 (intentionally or not) relies on the order of elements in the array returned by the method

Class.getDeclaredConstructors()

and under the Sun JRE the order happens to be “right”. However, the Java SE API specification says “…The elements in the array returned are not sorted and are not in any particular order.”

The full transcript of the root cause analysis of that issue is available here. We have also posted a comment to the JBoss issue tracker.

Now one may say: I don’t care about it. I deployed, deploy, and will deploy my application with the Sun JRE whatever other Java implementations would offer. OK, no problem but…

Kind of magic

Consider this simple program that reflects the declared constructors and prints the result:


import java.lang.reflect.*;

class Test{
    Test()         {}
    Test(Object o) {}
    Test(String s) {}

    public static void main(String args[]){
        for (Constructor c: Test.class.getDeclaredConstructors())
            System.out.println(c);
    }
}

If run on the Sun JRE 6, it prints:

    Test()
    Test(java.lang.Object)
    Test(java.lang.String)

So far so good. It looks like the constructors appear in the order they are declared in the source code. Let?s add a few instance methods to the code to make it a little bit more realistic:


    void dont() {}
    void rely() {}
    void on  () {}
    void it  () {}

and run the program on the same version of Sun JRE. Now it prints:

    Test(java.lang.String)
    Test(java.lang.Object)
    Test()

Surprise! If you want to find the rationale behind this behavior, you may dig into the OpenJDK sources starting from the method sort_methods() in the file hotspot\src\share\vm\runtime\classfileparser.cpp.

The lesson learned: Java applications may not rely on JVM features that are not enforced by the Java specification.

On speculative assumptions

Some observant readers may notice that the order of constructors from the examples above matches either direct or reverse order of their declaration in the source code. I however would not recommend to exploit this observation. Just add one more method:


    void ever() {}

to the code and run it to see yet another (different) order of the reflected constructors!

I must say that I did not get the last example out of my head. The quite popular JNA library used to rely on exactly this assumption regarding the method Class.getDeclaredFields(): the order of reflected fields was deemed either direct or reverse. Fortunately, in JNA 3.0.5, ?support for JVMs with unpredictable order of fields? has been added though I would rather call it ?improved compliance with the Java specification?.

Conclusion

Neglecting the Java spec, you still have a chance to get a working application. But you also have a chance that it will stop working if you modify the code or upgrade the underlying JVM. As a result, you may have to spend (waste) time on hunting subtle bugs or, even worse, get stuck with an outdated Java version if such bugs lurk in third-party components (yeah, J2SE 1.4.2 still has its ?thankful? audience).

What would be a solution for this problem? I cannot imagine a ?rule checker? that tests Java applications for compliance with the Java spec. Implementing such a tool is unlikely possible because the semantics of programs is involved.

For now, the only practical option is testing your Java application on different JVMs provided they support the same Java version (with the same level of Java compatibility). If the application does not work on one or more of them, it may be an issue worth investigating.

P.S. By the way, another large cluster of latent bugs that this testing approach can help you reveal is data races (aka random features) in multi-thread Java applications but that is another tale.

References

  1. Root cause analysis confirming the claim
  2. Excelsior JET JVM
  3. Java SE 6 API specification

Categories: Excelsior JET

10 Responses

  1. Dimitris Andreadis Says:

    Thanks for the report.

    I’ve seen this before in testsuite execution where the order of the junit tests would vary depending on the jdk used, unless you’ve declared them explicitly.

    I’m sure though we did test with JRockit, as we have a continuous hudson job for that.

  2. Tom Says:

    I have seen differences with field member order with the same JVM. The only difference was in using or not the debugger.

    For this reason, I think thank compile the JBOSS with the Sun JVM can have the same problems.

    As the post said:

    However, the Java SE API specification says ??The elements in the array returned are not sorted and are not in any particular order.

    Can I asume that a debugging program must be equal than a executing JAVA program? An interesting question.

  3. Simon Says:

    @Tom – it’s quite plausible that a debugger might have some effect. Running with debug enabled might, for example, impact the value of a hash that affects the order things are returned in..

  4. Vitaly Mikheev Says:

    @Dimitris

    Thank you for resolving the issue so quicky. Now we can avoid downgrading our test suite. ;)

    BTW, I recall I saw an announcement (about a year ago) that RedHat would put JBoss on the OSGi rails. Is there any progress in it?

  5. Vitaly Mikheev Says:

    @ Tom

    >>Can I asume that a debugging program must be equal
    >>than a executing JAVA program? An interesting question.

    It depends on what you mean under “equal”.

    Of course, in the debug mode when JVMTI is used, you run less optimized code. Otherwise you just could not inspect local variables, place break points on a particular line of code and so on.

    As a result, the thread scheduling picture may vary provoking various timing effects. It’s not a rare case, for example, that under debugger the (otherwise failing) application works.

    If that happens, it’s time to double check whether the data shared b/w multiple threads, are properly synchronized.

    ;)

  6. Rhys Parsons Says:

    I once had a similar problem with Java code running on a SCO box.

    The problem was with code that relied on ConnectException being thrown when a Socket failed to connect. Interesetingly, the JavaDoc for Socket doesn’t mention the ConnectException at all. The SCO JRE was throwing the more generic IOException, which the JavaDoc does mention.

    To this day I don’t know whether the SCO JRE was to blame or the code that assumed ConnectException would be thrown.

  7. Juergen Says:

    Please note that there are platforms (IBM’s) for which there is no Sun JDK: AIX, z/OS

  8. Vitaly Mikheev Says:

    @Juergen

    >>Please note that there are platforms (IBM?s) for which
    >>there is no Sun JDK: AIX, z/OS

    Good point. It reminds me the situation when under a consulting agreement we ported JRE to a new platform (not IBM?s though ;) ). There was no Java SE on that platform and I personally felt some kind of discomfort in the absence of the reference JRE. Of course, JCK is mighty but it does not cover every aspect of a Java implementation.

    But if you are talking about a pure Java application (I guess Java EE one), wouldn?t it be possible to run (a part of) the application on another JVM/OS?

  9. Peter Says:

    Recently i read a very interesting post regarding JVMs. i think it should be interesting for you guys so take a look.

  10. Peter Says:

    http://bigdatamatters.com/bigdatamatters/2009/08/jvm-performance.html