Excelsior Logo Home
Buy   
Download   
Support   
 Forum   
Blog   

A Java Compiler Performance Study

By Jan-Jaap van Horssen
August 2008

This case study describes the use of a draughts playing program to benchmark Java compilers. Two ahead-of-time Java to native compilers are benchmarked, as well as three versions of the Java HotSpot Virtual Machine using just-in-time compiling. All benchmarks were run on three different computers, with 32/64-bit Intel and AMD chips, all running versions of Windows XP.

Compilers

Although today's Java virtual machines are very efficient, using an optimizing ahead-of-time compiler might still improve performance. Another advantage of using a compiler is that your application becomes independent of the Java runtime environment (JRE). Instead, a dedicated runtime library is used.

The following Java to native compilers were used to compile and run the benchmark:

  1. JavaNativeCompiler (version 1.1.1), a graphical front-end for the Gnu Java compiler (GCJ as part of gcc, version 4.3.0)
  2. Excelsior JET (version 6.4)

Both compiled the benchmark JAR file into a Windows 32-bit executable.
Furthermore, the benchmark JAR was run on the following Java virtual machines:

  1. HotSpot Client  VM 32-bits (JRE  version 1.6.0)
  2. HotSpot Server VM 32-bits (JDK version 1.6.0)
  3. HotSpot Server VM 64-bits (JDK version 1.6.0) – when applicable/available

Assertions were disabled, by default.

Compatibility issues

Using the JavaNativeCompiler, I ran into a number of problems:

  • It supports the Java 5 language, but does not completely cover the Java 5 API. For the benchmark, I had to remove some (non-vital) code using the Scanner class and the String.format() method.
  • Unfortunately the NetBeans 6.1 GUI Builder I use generates some Java 6 code (for Free Design layout it uses the GroupLayout class). This means I had to change part of the GUI too.
  • After compilation was successful, the executable could not always locate a resource file residing inside the JAR. This was solved by moving the file outside the JAR.

Using the Excelsior JET compiler gave no problems at all. The benchmark compiled without errors or warnings. Full Java 6 including API is supported, even the non-standard sun.audio package. It offers many options and comes with a convenient utility for creating installation packages. This is required if you want your application to run on another machine.

Target platforms

All compiler benchmarks ran on three different computers:

Type CPU Word Size Clock Rate RAM Operating System
Laptop AMD Athlon 3000+ 64-bit 792 MHz 512 MB Windows XP 2002, SP2
Laptop Intel T2050 (duo) 32-bit 1.60 GHz 1.49 GB Windows XP 2002, SP2
Desktop Intel Core Duo E8400 64-bit 3.00 GHz 4 GB Windows XP pro x64 2003, SP1

For the AMD laptop no 64-bit HotSpot VM is available, so only the 32-bit versions were used. This leaves a total of 5 x 3 – 2 = 13 runs.

The benchmark program

The program used as a benchmark is ABCdam, a (Dutch) draughts playing program written in Java. It plays using the "international" rules for a 10x10 board, so it does not play checkers which is played on an 8x8 board.

For a draughts playing program, performance (speed) is of vital importance. Before choosing a move, it examines as many future positions as possible, given a limited amount of time.

Having done a lot of C and some C++ programming in the past, nowadays I prefer Java as a language. But how about Java performance?
Opinions on the internet differ. After a brief look at JNI (the Java native interface to C), I decided first to do some micro benchmarking with Java and C. Comparing the HotSpot Client VM to Gnu/Cygwin C showed Java to be as fast as compiled C code. It is of course always tricky to compare optimizing compilers like this, and a micro benchmark is not a real application – but still. So I decided to go for 100% Java.

The graphical user interface is developed using Swing/AWT and the NetBeans GUI Builder. For the performance critical sections, time consuming Java features are avoided (mainly object creation and garbage collection). Basically this means some C style programming where Java offers an easier solution. However, it saves significant runtime still allowing for an object oriented design.

The program is designed with a 64-bit CPU architecture in mind, so on a 64-bit platform it will run relatively faster – provided that the compiler or VM facilitates this. Before benchmarking, the program was optimized using the NetBeans profiler.

The benchmark

The benchmark consists of simulating a draughts game. A “standard” draughts game of 79 moves (for each side) is replayed, letting the program calculate a move in each position as if it were the player to move. Except for the forced capture moves which are played immediately, it is allowed to “think” 15 seconds for each move. This gives a constant running time of 34 minutes for all benchmarks on all platforms. Performance is measured by keeping score of the average number of positions examined per second.

From an application point of view it is a process using maximum CPU and a lot of RAM (200 MB for the benchmark), having almost no disk– and screen–I/O.

Results

All 13 benchmarks were run three times, to see if the results were consistent. The final scores are the averages of these runs, and are shown in the graph below. Individual results were usually close to the average (±1%), except on the desktop computer which showed ±8% for the JavaNativeCompiler, the others varying up to 4%.

The results show a similar pattern on all (effectively) 32-bit platforms. The Client and Server versions of the HotSpot VM are doing almost equally well. Compared to that, the Excelsior JET executable is about 20% faster. The JavaNativeCompiler executable, on the other hand, is at least 20% slower than the HotSpot VM. I did not use any custom GCJ compiler flags, so maybe this can be improved.

On the Windows XP pro x64 platform, the 64-bit HotSpot VM still wins. 64-bit versions of the native compilers are not yet available.

As for startup time (initializations and reading a large file from disk) the Excelsior JET executable was fastest on all platforms. This was not part of the benchmark.

C versus Java

Although not the topic of this case study, for me personally it was an open question whether Java would offer sufficient performance for an application like this, compared to C. Since I did not want to rewrite my Java code into C for a real comparison, I compared the search depth (which is a rough equivalent of speed) of ABCdam to other draughts playing programs, written in C. Even when spending more time analyzing each position (thus examining less positions) it usually reaches the same search depth as its opponents.

With (optimal) C theoretically being faster (and requiring less memory), I think Java proves to be a serious candidate even for applications requiring high CPU power and a lot of RAM. Even more so with the availability of good optimizing native compilers.

Links

Home | Company | Products | Services | Resources | Contact

Store | Downloads | Support | Forum | Blog | Sitemap

© 1999-2013 Excelsior LLC. All Rights Reserved.