|
|||||||||||||||||||||||||||||||||||
A Java Compiler Performance Study
By Jan-Jaap van Horssen This case study describes the use of a draughts playing program to benchmark Java compilers. Two ahead-of-time Java to native compilers are benchmarked, as well as three versions of the Java HotSpot Virtual Machine using just-in-time compiling. All benchmarks were run on three different computers, with 32/64-bit Intel and AMD chips, all running versions of Windows XP. CompilersAlthough today's Java virtual machines are very efficient, using an optimizing ahead-of-time compiler might still improve performance. Another advantage of using a compiler is that your application becomes independent of the Java runtime environment (JRE). Instead, a dedicated runtime library is used. The following Java to native compilers were used to compile and run the benchmark:
Both compiled the
benchmark JAR file into a Windows 32-bit executable.
Assertions were disabled, by default. Compatibility issuesUsing the JavaNativeCompiler, I ran into a number of problems:
Using the Excelsior JET compiler gave no problems at all. The benchmark compiled without errors or warnings. Full Java 6 including API is supported, even the non-standard sun.audio package. It offers many options and comes with a convenient utility for creating installation packages. This is required if you want your application to run on another machine. Target platformsAll compiler benchmarks ran on three different computers:
For the AMD laptop no 64-bit HotSpot VM is available, so only the 32-bit versions were used. This leaves a total of 5 x 3 – 2 = 13 runs. The benchmark programThe program used as a benchmark is ABCdam, a (Dutch) draughts playing program written in Java. It plays using the "international" rules for a 10x10 board, so it does not play checkers which is played on an 8x8 board.
For a draughts playing
program, performance (speed) is of vital importance. Before choosing
a move, it examines as many future positions as possible, given a
limited amount of time. The program is designed with a 64-bit CPU architecture in mind, so on a 64-bit platform it will run relatively faster – provided that the compiler or VM facilitates this. Before benchmarking, the program was optimized using the NetBeans profiler. The benchmarkThe benchmark
consists of simulating a draughts game. A “standard”
draughts game of 79 moves (for each side) is replayed, letting the
program calculate a move in each position as if it were the player to
move. Except for the forced capture moves which are played
immediately, it is allowed to “think” 15 seconds for each
move. This gives a constant running time of 34
minutes for all benchmarks on all
platforms. Performance is measured by keeping score of the average
number of positions examined per second. ResultsAll 13 benchmarks were run three times, to see if the results were consistent. The final scores are the averages of these runs, and are shown in the graph below. Individual results were usually close to the average (±1%), except on the desktop computer which showed ±8% for the JavaNativeCompiler, the others varying up to 4%.
The results show a similar pattern on all (effectively) 32-bit platforms. The Client and Server versions of the HotSpot VM are doing almost equally well. Compared to that, the Excelsior JET executable is about 20% faster. The JavaNativeCompiler executable, on the other hand, is at least 20% slower than the HotSpot VM. I did not use any custom GCJ compiler flags, so maybe this can be improved. On the Windows XP pro x64 platform, the 64-bit HotSpot VM still wins. 64-bit versions of the native compilers are not yet available. As for startup time (initializations and reading a large file from disk) the Excelsior JET executable was fastest on all platforms. This was not part of the benchmark. C versus JavaAlthough not the topic of
this case study, for me personally it was an open question whether
Java would offer sufficient performance for an application like this,
compared to C. Since I did not want to rewrite my Java code into C
for a real comparison, I compared the search depth (which is a rough
equivalent of speed) of ABCdam to other draughts playing programs,
written in C. Even when spending more time analyzing each position
(thus examining less positions) it usually reaches the same search
depth as its opponents. Links |
||||||||||||||||||||||||
 | ||||||||||||||||||||||||
|
Home | Company | Products | Services | Resources | Blog | Contact Store | Downloads | Support | Forum | Blog | Sitemap © 1999-2012 Excelsior LLC. All Rights Reserved. |
