The Volano Report

• Last updated on January 22, 2000. The most recent Volano Report is available here.

The tests presented in this report look at the performance and network scalability of 17 Java virtual machines on 7 operating systems using a common Intel hardware platform running VolanoMark™ 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:

1. Is it fast?
2. Does it scale?

For background information and download locations, please see the Java Benchmarks page on Volano's Web site. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments, along with a table showing the compiler and threading model of each Java platform.

Is it fast?

Run rules

The performance test was executed with the following commands on a local loop-back connection, using the heap size options shown below where possible:

Server

java -ms8m -mx64m COM.volano.Main

Client

java -ms8m -mx64m COM.volano.Mark -count 100

The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

Java Platform

Score

Tower TowerJ 3.1.4 Linux (a) IBM JDK 1.1.8 Windows NT IBM JDK 1.1.8 OS/2 Microsoft VM 3234 Windows NT Microsoft VM 3229 Windows 2000 IBM JDK 1.1.8 Linux Sun HotSpot Server 2.0 Windows NT Sun JDK 1.3 Beta Windows NT Sun JDK 1.2.2 Windows NT Sun JDK 1.2.1_04 Solaris Novell JDK 1.1.7 NetWare (b) Sun JDK 1.2.1_03 Solaris Inprise JDK 1.2.2 RC1 Linux Blackdown JDK 1.2.2 RC3 Linux Transvirtual Kaffe 1.0.5 Linux (c) Blackdown JDK 1.1.7 Linux JDK 1.1.8 FreeBSD

2309 2279 2136 2029 1970 1780 1669 1550 1485 1358 1229 1051 1039 943 908 285 173

Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform	Operating System	Results	Score
Tower TowerJ 3.1.4 Linux (a)	Red Hat Linux 6.0 Intel	2303, 2313, 2313, 2300	2309
IBM JDK 1.1.8 Windows NT	Windows NT Workstation 4.0	2282, 2276, 2280, 2282	2279
IBM JDK 1.1.8 OS/2	OS/2 Warp Server for e-business	2145, 2141, 2118, 2150	2136
Microsoft VM 3234 Windows NT	Windows NT Workstation 4.0	2034, 2040, 2030, 2018	2029
Microsoft VM 3229 Windows 2000	Windows 2000 Server Release Candidate 2	1948, 1965, 1976, 1968	1970
IBM JDK 1.1.8 Linux	Red Hat Linux 6.0 Intel	1317, 1823, 1766, 1752	1780
Sun HotSpot Server 2.0 Windows NT	Windows NT Workstation 4.0	1638, 1664, 1668, 1675	1669
Sun JDK 1.3 Beta Windows NT	Windows NT Workstation 4.0	1538, 1550, 1556, 1545	1550
Sun JDK 1.2.2 Windows NT	Windows NT Workstation 4.0	1496, 1481, 1489, 1485	1485
Sun JDK 1.2.1_04 Solaris	Solaris 7 Desktop Intel Platform Edition	1371, 1344, 1374, 1355	1358
Novell JDK 1.1.7 NetWare (b)	NetWare 5	1227, 1232, 1229, 1226	1229
Sun JDK 1.2.1_03 Solaris	Solaris 7 Desktop Intel Platform Edition	1118, 1055, 1050, 1049	1051
Inprise JDK 1.2.2 RC1 Linux	Red Hat Linux 6.0 Intel	1032, 1040, 1036, 1040	1039
Blackdown JDK 1.2.2 RC3 Linux	Red Hat Linux 6.1 Intel	939, 945, 942, 943	943
Transvirtual Kaffe 1.0.5 Linux (c)	Red Hat Linux 6.0 Intel	906, 905, 909, 909	908
Blackdown JDK 1.1.7 Linux	Red Hat Linux 6.0 Intel	284, 285, 286, 283	285
JDK 1.1.8 FreeBSD	FreeBSD 3.2-RELEASE	169, 173, 173, 172	173

Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the last three results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Notes:

1. TowerJ is the only Java platform listed here which is not available for free.
2. The JVM for NetWare reported "java.net.SocketException: Software caused connection abort" once and "java.net.SocketException: Socket closed" 42 times when the server side was killed.
3. The Kaffe OpenVM failed on the first test run with "Kaffe: exception.c:308: dispatchException: Assertion `!intsDisabled()' failed." Note that the Kaffe OpenVM is the only Java platform listed here which is still unable to run our VolanoChat product due to Kaffe Bug io/78, "java.io.StreamCorruptedException".

Does it scale?

Run rules

The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection, using the heap and stack size options shown below where possible:

Server

java -ms8m -mx128m -ss32k COM.volano.Main

Client

java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200

Note: IBM JDK 1.1.8 on Linux required an initial heap size of 64 MB (using the option -ms64m) in order to complete the network scalability tests shown in Table 2 below, so you should judge its relative performance instead by its loopback score in Table 1 above.

See the COM.volano.Mark command synopsis for a complete description of all options.

The client test driver was executed under the Sun JDK 1.2.1_03 Production Release for Solaris using Solaris 7 on a dual 200-MHz Sun UltraSPARC with 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

1000	2000	3000	4000
Tower TowerJ 3.1.4 Linux IBM JDK 1.1.8 Windows NT IBM JDK 1.1.8 OS/2 Microsoft VM 3234 Windows NT Microsoft VM 3229 Windows 2000 IBM JDK 1.1.8 Linux Sun HotSpot Server 2.0 Windows NT Sun JDK 1.3 Beta Windows NT Sun JDK 1.2.2 Windows NT Sun JDK 1.2.1_04 Solaris Novell JDK 1.1.7 NetWare Sun JDK 1.2.1_03 Solaris Inprise JDK 1.2.2 RC1 Linux Blackdown JDK 1.2.2 RC3 Linux Transvirtual Kaffe 1.0.5 Linux Blackdown JDK 1.1.7 Linux JDK 1.1.8 FreeBSD

Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform	1000	2000	3000	4000	Notes
Tower TowerJ 3.1.4 Linux	2486	1357	739	421	No errors!
IBM JDK 1.1.8 Windows NT	2623	1826	1081	----	"java.net.SocketException: Socket read failed: 10107" at 3553/4000 connections.
IBM JDK 1.1.8 OS/2	1521	----	----	----	"java.lang.OutOfMemoryError" at 1098/2000 connections.
Microsoft VM 3234 Windows NT	1695	979	535	----	"java.lang.OutOfMemoryError" at 3796/4000 connections.
Microsoft VM 3229 Windows 2000	1408	708	406	----	"java.lang.OutOfMemoryError" at 3785/4000 connections.
IBM JDK 1.1.8 Linux	722	556	----	----	"java.lang.OutOfMemoryError: cannot create any more threads" at 2007/3000 connections. Required an initial heap size of 64 MB instead of 8 MB.
Sun HotSpot Server 2.0 Windows NT	----	----	----	----	"HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION, Error ID: 4F533F57494E13120E43505002D9" at 880/1000 connections.
Sun JDK 1.3 Beta Windows NT	1474	856	523	----	"java.lang.OutOfMemoryError" at 3272/4000 connections.
Sun JDK 1.2.2 Windows NT	----	----	----	----	"java.lang.OutOfMemoryError" at 926/1000 connections.
Sun JDK 1.2.1_04 Solaris	1875	1040	659	----	"Bus error - core dumped" on 4000-connection test.
Novell JDK 1.1.7 NetWare	1575	----	----	----	Process hangs with disk thrashing at 1960/2000 connections.
Sun JDK 1.2.1_03 Solaris	1794	1058	613	354	No errors!
Inprise JDK 1.2.2 RC1 Linux	----	----	----	----	Fails with "java.net.SocketException: Interrupted system call". See Bug Id 4298809.
Blackdown JDK 1.2.2 RC3 Linux	1153	----	----	----	Fails the 2000-connection test with "SIGSEGV 11* segmentation violation" at just over 1000 connections. See Bug Id 1604.
Transvirtual Kaffe 1.0.5 Linux	----	----	----	----	Hangs with zero percent CPU usage after starting, even at just 500 connections. Dumps core when the client side is killed to stop the test.
Blackdown JDK 1.1.7 Linux	165	----	----	----	Dumps core at just over 1000 connections.
JDK 1.1.8 FreeBSD	37	----	----	----	Client begins failing at 1017 connections with "java.net.NoRouteToHostException: Connection timed out".

Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Environments

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware -- a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM on an Intel VS440FX motherboard running American Megatrends AMIBIOS Version 1.00.18.CS1 dated "08/28/98-13:59". The network card is a 3Com Etherlink III PCI Bus-Master Adapter Model 3C590.

Tower TowerJ 3.1.4 Linux

• Tower Technology TowerJ High Performance Java Compiler Version 3.1.4.0
• Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
• TowerJ Compiler "version 3.1.4.0 x86-linux"
• Installed from TowerJ_3_1_4_0_x86_linux.class (12,913,370 bytes).
• Uses user-level threads and static native pre-compilation.
• Used the command line option -Dtowerj.poll for the network scalability test.
• Built executable with TowerJ project files Main.tj and Mark.tj.
• Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.

IBM JDK 1.1.8 Windows NT

• IBM Developer Kit and Runtime Environment for Windows, Java Technology Edition, Version 1.1.8
• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
• Java version "JDK 1.1.8 IBM build n118p-19990728 (JIT enabled: ibmjitc V3.5-IBMJDK1.1-19990728)"
• Installed from ibm-jdk-n118p-win32-x86.zip (10,810,316 bytes).
• Uses native threads and ibmjitc just-in-time compiler.

IBM JDK 1.1.8 OS/2

• IBM OS/2 Warp Developer Kit, Java Technology Edition, Version 1.1.8
• IBM OS/2 Warp Server for e-business (Version 20.45, Internal revision 14.039F_UNI)
• Java version "JDK 1.1.8 IBM build o118-19990728 (JIT enabled: javax V3.5-IBMJDK1.1-19990728)"
• Installed from javainuf.exe (20,805,936 bytes) and javaintk.exe (15,273,452 bytes).
• Uses native threads and javax just-in-time compiler.
• Modified the CONFIG.SYS file to increase the thread limit from 1024 to 4095 (THREADS=4095) and the initial swap file size from 2 MB to 32 MB (SWAPPATH=C:\OS2\SYSTEM 2048 32768).

Microsoft VM 3234 Windows NT

• Microsoft Virtual Machine Build 3234
• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
• Java version "1.1.4"
• jview version 5.00.3229
• msjava.dll version 5.0.3234.0
• Installed from msjavx86.exe (5,449,664 bytes).
• Uses native threads and just-in-time compiler.
• Heap and stack command line options are not available.

Microsoft VM 3229 Windows 2000

• Microsoft Virtual Machine Build 3229
• Microsoft Windows 2000 Server Release Candidate 2 (Version 5.0.2128)
• Java version "1.1.4"
• jview version 5.00.3229
• Uses native threads and just-in-time compiler.
• Heap and stack command line options are not available.

IBM JDK 1.1.8 Linux

• IBM Developer Kit and Runtime Environment for Linux, Java Technology Edition, Version 1.1.8
• Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
• Java version "JDK 1.1.8 IBM build l118-19991221 (JIT enabled: jitc)"
• Installed from ibm-jdk-l118-linux-x86.tgz (10,560,355 bytes).
• Uses native threads and jitc just-in-time compiler.
• Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.
• Ran the network scalability test with an initial heap size of 64 MB instead of 8 MB in order for it to work. With an initial heap size of 8 MB, the performance was too poor to complete the 2,000 connection test without having the clients receive timeout errors.

Sun HotSpot Server 2.0 Windows NT

• Sun Java HotSpot Server VM Version 2.0 Beta for Win32 Platforms
• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
• Sun JDK base version "Classic VM (build JDK-1.2.2-001, native threads, symcjit)"
• Sun HotSpot version "Java HotSpot(TM) Server VM (2.0rc1, mixed mode, build I)"
• Installed from jdk1_2_2-001-win.exe (20,449,862 bytes) and hotspot2_0rc1-win.exe (1,305,731 bytes).
• Uses native threads and an adaptive compiler.
• The heap options must be preceded by the capital letter "X".

Sun JDK 1.3 Beta Windows NT

• Sun Java 2 Software Development Kit, Standard Edition, v.1.3 Beta
• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
• Java version "1.3beta, Java(TM) 2 Runtime Environment, Standard Edition (build 1.3beta-0), Java(TM) HotSpot Client VM (build 1.3beta-0, mixed mode)"
• Installed from jdk1_3beta-win.exe (25,662,367 bytes).
• Uses native threads and an adaptive compiler.
• The heap options must be preceded by the capital letter "X", and the stack options are not available.

Sun JDK 1.2.2 Windows NT

• Sun Java 2 SDK v1.2.2 Windows 95/98/NT Production Release
• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 5)
• Java version "Classic VM (build JDK-1.2.2-W, native threads, symcjit)"
• Installed from jdk1_2_2-win.exe (20,422,291 bytes).
• Uses native threads and the symcjit Symantec Just-in-Time compiler.
• The heap options must be preceded by the capital letter "X", and the stack options are not available.

Sun JDK 1.2.1_04 Solaris

• Sun Java 2 SDK Solaris Production Pre-Release
• Sun Solaris 7 Desktop Intel Platform Edition with Maintenance Update 1 and patches 106981-05, 107079-10, 107608-01, and 107637-01.
• Java version "Solaris_JDK_1.2.1_04_pre-release"
• Installed from Solaris_JDK_1.2.1_04_pre-release_i386.bin (18,607,361 bytes).
• Uses native threads and sunwjit just-in-time compiler.
• The heap and stack options must be preceded by the capital letter "X".
• Increased the per-process file descriptor limit to 8192 by setting the rlim_fd_max variable in /etc/system.

Novell JDK 1.1.7 NetWare

• Novell JVM for NetWare
• Novell NetWare 5 Support Pack 2 Version 5.00c dated April 23, 1999
• Java version "1.1.7 B"
• Installed from jvm.exe (18,015,477 bytes).
• Uses native threads and symcjit just-in-time compiler (Symantec Java! JustInTime Compiler Version 3.10.106(x) for JDK 1.1.x).
• Set Maximum Packet Receive Buffers = 1000 (from default of 500).
• Modified the java.cfg file to enable the just-in-time compiler (JAVA_COMPILER=symcjit).

Sun JDK 1.2.1_03 Solaris

• Sun Java 2 SDK Solaris Production Release
• Sun Solaris 7 Desktop Intel Platform Edition with patches 106981-04 and 107079-08.
• Java version "Solaris_JDK_1.2.1_03"
• Installed from 1.2.1_03_jdk_i386.tar.Z (26,264,747 bytes).
• Uses native threads and sunwjit just-in-time compiler.
• The heap and stack options must be preceded by the capital letter "X".
• Increased the per-process file descriptor limit to 8192 by setting the rlim_fd_max variable in /etc/system.

Inprise JDK 1.2.2 RC1 Linux

• Java 2 SDK, Standard Edition, v 1.2.2, Release Candidate, Linux Port
• Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
• Java version "1.2.2", "Classic VM (build 1.2.2-I, green threads, javacomp)"
• Installed from jdk1_2_2rc1-linux-i386.tar.gz (21,794,070 bytes).
• Uses green threads and javacomp just-in-time compiler.
• The heap and stack options must be preceded by the capital letter "X".
• Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096).

Blackdown JDK 1.2.2 RC3 Linux

• Blackdown Java-Linux port of the Java 2 SDK Version 1.2.2 Release Candidate 3
• Red Hat Linux Intel 6.1 (Linux 2.2.12-20, glibc 2.1.2-11)
• Java version "1.2.2", "Classic VM (build Linux_JDK_RC3, green threads, sunwjit)"
• Installed from jdk-1.2.2-RC3-linux-i386.sh (21,028,547 bytes).
• Uses native or green threads and sunwjit just-in-time compiler.
• Used the "-green" option for user-level threads.
• The heap and stack options must be preceded by the capital letter "X".
• Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096).

Transvirtual Kaffe 1.0.5 Linux

• Transvirtual Technologies Kaffe OpenVM Version 1.0.5
• Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
• Kaffe Virtual Machine "Engine: Just-in-time v3 Version: 1.0.5 Java Version: 1.1"
• Installed from the kaffe-1.0.5.tar.gz (3,145,844 bytes).
• Uses user-level threads and kaffe.jit just-in-time compiler.
• Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.

Blackdown JDK 1.1.7 Linux

• Blackdown Java-Linux port of JDK 1.1.7
• Red Hat Linux Intel 6.0 (Linux 2.2.5-15, glibc 2.1.1-6)
• Java version "Linux_JDK_1.1.7B_v3_green_threads"
• Installed from jdk_1.1.7-v3-glibc-x86.tar.bz2 (10,130,082 bytes).
• Uses user-level threads and no just-in-time compiler.
• Increased the per-process file descriptor limit to 4096 (from 1024) and the system-wide file descriptor limit to 8192 (from 4096). Increased the maximum tasks per user to 4090 (from 256) and the system-wide task limit to 4090 (from 512). Compiled the Linux kernel for the "PPro/6x86MX" processor family.

JDK 1.1.8 FreeBSD

• JDK 1.1.8 for FreeBSD
• FreeBSD 3.2-RELEASE
• Java version "jdk1.1.8-FreeBSD:1999/7/19"
• Installed from jdk1.1.8_ELF.V99-7-19.tar.gz (11,337,773 bytes).
• Uses user-level threads and no just-in-time compiler.
• Increased the per-process file descriptor limit to 4096 (from 1064) and the system-wide file descriptor limit to 8192 (from 1064) by using sysctl to modify the kern.maxfilesperproc and kern.maxfiles variables in /etc/rc.local.

Java Platform	Operating System	Compiler	Thread Model
Tower TowerJ 3.1.4 Linux	Red Hat Linux 6.0 Intel	static	many-to-one
IBM JDK 1.1.8 Windows NT	Windows NT Workstation 4.0	mixed mode	one-to-one
IBM JDK 1.1.8 OS/2	OS/2 Warp Server for e-business	mixed mode	one-to-one
Microsoft VM 3234 Windows NT	Windows NT Workstation 4.0	JIT	one-to-one
Microsoft VM 3229 Windows 2000	Windows 2000 Server Release Candidate 2	JIT	one-to-one
IBM JDK 1.1.8 Linux	Red Hat Linux 6.0 Intel	JIT	one-to-one
Sun HotSpot Server 2.0 Windows NT	Windows NT Workstation 4.0	mixed mode	one-to-one
Sun JDK 1.3 Beta Windows NT	Windows NT Workstation 4.0	mixed mode	one-to-one
Sun JDK 1.2.2 Windows NT	Windows NT Workstation 4.0	JIT	one-to-one
Sun JDK 1.2.1_04 Solaris	Solaris 7 Desktop Intel Platform Edition	JIT	many-to-many
Novell JDK 1.1.7 NetWare	NetWare 5	JIT	many-to-one
Sun JDK 1.2.1_03 Solaris	Solaris 7 Desktop Intel Platform Edition	JIT	many-to-many
Inprise JDK 1.2.2 RC1 Linux	Red Hat Linux 6.0 Intel	JIT	many-to-one
Blackdown JDK 1.2.2 RC3 Linux	Red Hat Linux 6.1 Intel	JIT	one-to-one or many-to-one
Transvirtual Kaffe 1.0.5 Linux	Red Hat Linux 6.0 Intel	JIT	many-to-one
Blackdown JDK 1.1.7 Linux	Red Hat Linux 6.0 Intel	none	many-to-one
JDK 1.1.8 FreeBSD	FreeBSD 3.2-RELEASE	none	many-to-one

Table 3. The compiler and threading model used by each Java platform.

Compiler

static

A static compiler which takes Java source files or class files and translates them into a native executable or shared library before they run.

mixed mode

A Java VM which translates some of the Java bytecodes into native instructions on the fly while the Java program runs, while letting a Java interpreter handle other parts of the program. Specifically, I use this term to refer to adaptive compilers which use runtime execution profiles to optimize the resulting native instructions.

JIT

A Java VM which translates the Java bytecodes into native instructions on the fly before executing them.

none

A Java VM which uses a normal Java bytecode interpreter.

Thread Model

one-to-one

Each Java thread is mapped one-to-one onto its own operating system thread.

many-to-one

All of the Java threads in the process are mapped onto one operating system process.

many-to-many

More than one Java thread is mapped to one operating system thread, and a Java thread may be mapped to different operating system threads during its lifetime.