A Java Compiler Performance Study
By Jan-Jaap van Horssen
This case study describes
the use of a draughts playing program to benchmark Java compilers.
Two ahead-of-time Java to native compilers are benchmarked, as well
as three versions of the Java HotSpot Virtual Machine using
just-in-time compiling. All benchmarks were run on three different
computers, with 32/64-bit Intel and AMD chips, all running versions
of Windows XP.
Java virtual machines are very efficient, using an optimizing
compiler might still improve performance. Another advantage of
using a compiler is that your application becomes independent of the
Java runtime environment (JRE). Instead, a dedicated runtime library
The following Java to
native compilers were used to compile and run the benchmark:
(version 1.1.1), a graphical front-end for the Gnu Java compiler
(GCJ as part of gcc, version 4.3.0)
JET (version 6.4)
Both compiled the
benchmark JAR file into a Windows 32-bit executable.
the benchmark JAR was run on the following Java virtual machines:
- HotSpot Client
VM 32-bits (JRE version 1.6.0)
- HotSpot Server VM
32-bits (JDK version 1.6.0)
- HotSpot Server VM
64-bits (JDK version 1.6.0) – when applicable/available
Assertions were disabled,
Using the JavaNativeCompiler, I ran into a number of problems:
- It supports the
Java 5 language, but does not completely cover the Java 5 API. For
the benchmark, I had to remove some (non-vital) code using the
class and the String.format()
Unfortunately the NetBeans 6.1 GUI Builder I use
generates some Java 6 code (for Free Design layout it uses the
class). This means I had to change part of the GUI too.
compilation was successful, the executable could not always locate a
resource file residing inside the JAR. This was solved by moving the
file outside the JAR.
Using the Excelsior JET compiler gave no
problems at all. The benchmark compiled without errors or warnings.
Full Java 6 including API is supported, even the non-standard
package. It offers many options and comes with a convenient utility
for creating installation packages. This is required if you want your
application to run on another machine.
All compiler benchmarks
ran on three different computers:
||AMD Athlon 3000+
||Windows XP 2002, SP2
||Intel T2050 (duo)
||Windows XP 2002, SP2
||Intel Core Duo E8400
||Windows XP pro x64 2003, SP1
For the AMD laptop no
64-bit HotSpot VM is available, so only the 32-bit versions were
used. This leaves a total of 5 x 3 – 2 = 13 runs.
The benchmark program
The program used as
a benchmark is ABCdam, a (Dutch) draughts
playing program written in Java. It plays using the
"international" rules for a 10x10 board, so it does not
play checkers which is played on an 8x8 board.
For a draughts playing
program, performance (speed) is of vital importance. Before choosing
a move, it examines as many future positions as possible, given a
limited amount of time.
Having done a lot of C and some C++
programming in the past, nowadays I prefer Java as a language. But
how about Java performance?
Opinions on the internet differ. After
a brief look at JNI (the Java native interface to C), I decided first
to do some micro benchmarking with Java and C. Comparing the HotSpot
Client VM to Gnu/Cygwin C showed Java to be as fast as compiled C
code. It is of course always tricky to compare optimizing compilers
like this, and a micro benchmark is not a real application –
but still. So I decided to go for 100% Java.
user interface is developed using Swing/AWT and the NetBeans GUI
Builder. For the performance critical sections, time consuming Java
features are avoided (mainly object creation and garbage collection).
Basically this means some C style programming where Java offers an
easier solution. However, it saves significant runtime still allowing
for an object oriented design.
The program is designed with a
64-bit CPU architecture in mind, so on a 64-bit platform it will run
relatively faster – provided that the compiler or VM
facilitates this. Before benchmarking, the program was optimized
using the NetBeans profiler.
consists of simulating a draughts game. A “standard”
draughts game of 79 moves (for each side) is replayed, letting the
program calculate a move in each position as if it were the player to
move. Except for the forced capture moves which are played
immediately, it is allowed to “think” 15 seconds for each
move. This gives a constant running time of 34
minutes for all benchmarks on all
platforms. Performance is measured by keeping score of the average
number of positions examined per second.
From an application
point of view it is a process using maximum CPU and a lot of RAM (200
MB for the benchmark), having almost no disk– and screen–I/O.
All 13 benchmarks were run
three times, to see if the results were consistent. The final scores
are the averages of these runs, and are shown in the graph below.
Individual results were usually close to the average (±1%),
except on the desktop computer which showed ±8% for the
JavaNativeCompiler, the others varying up to 4%.
The results show a similar
pattern on all (effectively) 32-bit platforms. The Client and Server
versions of the HotSpot VM are doing almost equally well. Compared to
that, the Excelsior JET executable is about 20% faster. The
JavaNativeCompiler executable, on the other hand, is at least 20%
slower than the HotSpot VM. I did not use any custom GCJ compiler
flags, so maybe this can be improved.
On the Windows XP pro x64
platform, the 64-bit HotSpot VM still wins. 64-bit versions of the
native compilers are not yet available.
As for startup time
(initializations and reading a large file from disk) the Excelsior
JET executable was fastest on all platforms. This was not part of the
C versus Java
Although not the topic of
this case study, for me personally it was an open question whether
Java would offer sufficient performance for an application like this,
compared to C. Since I did not want to rewrite my Java code into C
for a real comparison, I compared the search depth (which is a rough
equivalent of speed) of ABCdam to other draughts playing programs,
written in C. Even when spending more time analyzing each position
(thus examining less positions) it usually reaches the same search
depth as its opponents.
With (optimal) C theoretically being
faster (and requiring less memory), I think Java proves to be a
serious candidate even for applications requiring high CPU power and
a lot of RAM. Even more so with the availability of good optimizing