The Volano Report

What's new?

Much to my surprise, Appeal's JRockit has arrived on the scene to post the best network scalability scores of any Java virtual machine I've tested. In fact, it's the only Java virtual machine I've found which is able to support 4,000 connections on Windows NT or Windows 2000 without modification.

Appeal has made Java threads as lightweight as they're advertised to be, mapping their thin threads onto native threads as required. JRockit seems to be particularly suited for pure Java servers which must handle many dedicated network connections. Like TowerJ, though, it's not free. Let's hope for a Linux port of JRockit!

What is VolanoMark?

For background information and download locations, please see the Java Benchmarks page on Volano's Web site. Since the server side of the VolanoMark benchmark is our VolanoChat server, these test results help our customers choose the best Java virtual machine for their environment.

VolanoMark is a pure Java server benchmark characterized by long-lasting network connections and high thread counts. In this context, long-lasting means the connections last several minutes or longer, rather than just a few seconds. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second.

Bug Id 4075058, "java.io: Add support for non-blocking I/O," explains why you need lots of threads in order to handle dedicated connections to a pure Java server. JSR 51, "New I/O APIs for the Java Platform," explains how Sun plans to fix that.

The tests

The tests presented in this report look at the performance and network scalability of 18 Java virtual machines on 7 operating systems using a common Intel hardware platform running VolanoMark 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:

  1. Is it fast?
  2. Does it scale?

See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments, along with a table showing the compiler and threading model of each Java platform.

Is it fast?

Run rules

The performance test was executed with the following commands on a local loop-back connection, using the heap size options shown below where possible:

Server
java -ms8m -mx64m COM.volano.Main
Client
java -ms8m -mx64m COM.volano.Mark -count 100

See the COM.volano.Mark command synopsis for a complete description of all options.

The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.

Results

Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

Java Platform Score
Tower TowerJ 3.1.4 Linux (a)
IBM JDK 1.1.8 Windows NT
IBM JDK 1.1.8 OS/2
Microsoft VM 3234 Windows NT
Microsoft VM 3229 Windows 2000
Appeal JRockit 1.0 Windows 2000 (a)
IBM JDK 1.1.8 Linux
Sun HotSpot Server 2.0 Windows NT
Sun JDK 1.3 RC1 Windows NT
Sun JDK 1.2.2-001 Windows NT
Sun JDK 1.2.1_04 Solaris
Novell JDK 1.1.7 NetWare (b)
Sun/Inprise JDK 1.2.2 Linux
Blackdown/Inprise JDK 1.2.2 Linux
Blackdown JDK 1.2.2 RC4 Linux
Transvirtual Kaffe 1.0.5 Linux (c)
Blackdown JDK 1.1.7 Linux
JDK 1.1.8 FreeBSD
2309 2309
2266 2266
2136 2136
2029 2029
1970 1970
1927 1927
1780 1780
1669 1669
1592 1592
1477 1477
1399 1399
1229 1229
1040 1040
1038 1038
952 952
908 908
285 285
173 173
Figure 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second.

Java Platform Operating System Results Score
Tower TowerJ 3.1.4 Linux (a) Red Hat Linux 6.0 Intel 2303, 2313, 2313, 2300 2309
IBM JDK 1.1.8 Windows NT Windows NT Workstation 4.0 2244, 2267, 2267, 2263 2266
IBM JDK 1.1.8 OS/2 OS/2 Warp Server for e-business 2145, 2141, 2118, 2150 2136
Microsoft VM 3234 Windows NT Windows NT Workstation 4.0 2034, 2040, 2030, 2018 2029
Microsoft VM 3229 Windows 2000 Windows 2000 Server Release Candidate 2 1948, 1965, 1976, 1968 1970
Appeal JRockit 1.0 Windows 2000 (a) Windows 2000 Server 1910, 1926, 1925, 1931 1927
IBM JDK 1.1.8 Linux Red Hat Linux 6.0 Intel 1317, 1823, 1766, 1752 1780
Sun HotSpot Server 2.0 Windows NT Windows NT Workstation 4.0 1638, 1664, 1668, 1675 1669
Sun JDK 1.3 RC1 Windows NT Windows NT Workstation 4.0 1570, 1593, 1578, 1605 1592
Sun JDK 1.2.2-001 Windows NT Windows NT Workstation 4.0 1477, 1481, 1479, 1471 1477
Sun JDK 1.2.1_04 Solaris Solaris 7 Desktop Intel Platform Edition 1409, 1408, 1400, 1389 1399
Novell JDK 1.1.7 NetWare (b) NetWare 5 1227, 1232, 1229, 1226 1229
Sun/Inprise JDK 1.2.2 Linux Red Hat Linux 6.0 Intel 1035, 1039, 1042, 1039 1040
Blackdown/Inprise JDK 1.2.2 Linux Red Hat Linux 6.0 Intel 1032, 1037, 1039, 1037 1038
Blackdown JDK 1.2.2 RC4 Linux Red Hat Linux 6.0 Intel 951, 952, 953, 950 952
Transvirtual Kaffe 1.0.5 Linux (c) Red Hat Linux 6.0 Intel 906, 905, 909, 909 908
Blackdown JDK 1.1.7 Linux Red Hat Linux 6.0 Intel 284, 285, 286, 283 285
JDK 1.1.8 FreeBSD FreeBSD 3.2-RELEASE 169, 173, 173, 172 173
Table 1. VolanoMark 2.1.2 local performance test, measuring throughput in messages per second. The final score is the average of the last three results. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Notes:

  1. TowerJ and JRockit are the only Java platforms listed here which are not available for free and are not intended for use with client Java applications.
  2. The JVM for NetWare reported "java.net.SocketException: Software caused connection abort" once and "java.net.SocketException: Socket closed" 42 times when the server side was killed.
  3. The Kaffe OpenVM failed on the first test run with "Kaffe: exception.c:308: dispatchException: Assertion `!intsDisabled()' failed."

Does it scale?

Run rules

The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection, using the heap and stack size options shown below where possible:

Server
java -ms8m -mx128m -ss32k COM.volano.Main
Client
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200

Note: IBM JDK 1.1.8 on Linux required an initial heap size of 64 MB (using the option -ms64m) in order to complete the network scalability tests shown in Table 2 below, so you should judge its relative performance instead by the loopback scores in Table 1 above.

See the COM.volano.Mark command synopsis for a complete description of all options.

The client test driver was executed under the Sun JDK 1.2.1_03 Production Release for Solaris using Solaris 7 on a dual 200-MHz Sun UltraSPARC with 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.

Results

Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.

2486 2571 1521 1695 1408 2354 722 1521 1821 1575 1174 1177 1141 165 37 1357 1755 979 708 1901 556 871 1060 718 666 739 1072 535 406 1237 591 695 473 452 421 688 310
1000 2000 3000 4000
Tower TowerJ 3.1.4 Linux
IBM JDK 1.1.8 Windows NT
IBM JDK 1.1.8 OS/2
Microsoft VM 3234 Windows NT
Microsoft VM 3229 Windows 2000
Appeal JRockit 1.0 Windows 2000
IBM JDK 1.1.8 Linux
Sun HotSpot Server 2.0 Windows NT
Sun JDK 1.3 RC1 Windows NT
Sun JDK 1.2.2-001 Windows NT
Sun JDK 1.2.1_04 Solaris
Novell JDK 1.1.7 NetWare
Sun/Inrpise JDK 1.2.2 Linux
Blackdown/Inprise JDK 1.2.2 Linux
Blackdown JDK 1.2.2 RC4 Linux
Transvirtual Kaffe 1.0.5 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.8 FreeBSD
Figure 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections.

Java Platform 1000 2000 3000 4000 Notes
Tower TowerJ 3.1.4 Linux 2486 1357 739 421 No errors!
IBM JDK 1.1.8 Windows NT 2571 1755 1072 ---- "java.lang.OutOfMemoryError" at 3508/4000 connections.
IBM JDK 1.1.8 OS/2 1521 ---- ---- ---- "java.lang.OutOfMemoryError" at 1098/2000 connections.
Microsoft VM 3234 Windows NT 1695 979 535 ---- "java.lang.OutOfMemoryError" at 3796/4000 connections.
Microsoft VM 3229 Windows 2000 1408 708 406 ---- "java.lang.OutOfMemoryError" at 3785/4000 connections.
Appeal JRockit 1.0 Windows 2000 2354 1901 1237 688 No errors!
IBM JDK 1.1.8 Linux 722 556 ---- ---- "java.lang.OutOfMemoryError: cannot create any more threads" at 2007/3000 connections. Required an initial heap size of 64 MB instead of 8 MB.
Sun HotSpot Server 2.0 Windows NT ---- ---- ---- ---- "HotSpot Virtual Machine Error, EXCEPTION_ACCESS_VIOLATION, Error ID: 4F533F57494E13120E43505002D9" at 880/1000 connections.
Sun JDK 1.3 RC1 Windows NT 1521 871 591 ---- "java.lang.OutOfMemoryError" at 3476/4000 connections.
Sun JDK 1.2.2-001 Windows NT ---- ---- ---- ---- "java.lang.OutOfMemoryError" at 926/1000 connections.
Sun JDK 1.2.1_04 Solaris 1821 1060 695 ---- Hangs with disk thrashing at 3940/4000 connections. "Segmentation Fault - core dumped" when client test driver is killed to stop the test.
Novell JDK 1.1.7 NetWare 1575 ---- ---- ---- Hangs with disk thrashing at 1960/2000 connections.
Sun/Inprise JDK 1.2.2 Linux 1174 ---- ---- ---- "SIGSEGV 11* segmentation violation" at just over 1000 connections. See the second of the two bugs placed into Sun Bug Id 4298809. Only the first bug has been fixed.
Blackdown/Inprise JDK 1.2.2 Linux 1177 718 473 310 No errors!
Blackdown JDK 1.2.2 RC4 Linux 1141 666 452 ---- "java.lang.StackOverflowError" after a total of 6816 cumulative connections, causing it to fail 816 connections into the 4000-connection test. See Sun Bug Id 4299454.
Transvirtual Kaffe 1.0.5 Linux ---- ---- ---- ---- Hangs with zero percent CPU usage after starting, even at just 500 connections. Dumps core when the client side is killed to stop the test.
Blackdown JDK 1.1.7 Linux 165 ---- ---- ---- Dumps core at just over 1000 connections.
JDK 1.1.8 FreeBSD 37 ---- ---- ---- Client begins failing at 1017 connections with "java.net.NoRouteToHostException: Connection timed out".
Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second based on the number of concurrent connections. All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.

Environments

All tests ran identical copies of VolanoMark 2.1.2 on identical hardware -- a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM on an Intel VS440FX motherboard running American Megatrends AMIBIOS Version 1.00.18.CS1 dated "08/28/98-13:59". The network card is a 3Com Etherlink III PCI Bus-Master Adapter Model 3C590.

Tower TowerJ 3.1.4 Linux
IBM JDK 1.1.8 Windows NT
IBM JDK 1.1.8 OS/2
Microsoft VM 3234 Windows NT
Microsoft VM 3229 Windows 2000
Appeal JRockit 1.0 Windows 2000
IBM JDK 1.1.8 Linux
Sun HotSpot Server 2.0 Windows NT
Sun JDK 1.3 RC1 Windows NT
Sun JDK 1.2.2-001 Windows NT
Sun JDK 1.2.1_04 Solaris
Novell JDK 1.1.7 NetWare
Sun/Inprise JDK 1.2.2 Linux
Blackdown/Inprise JDK 1.2.2 Linux
Blackdown JDK 1.2.2 RC4 Linux
Transvirtual Kaffe 1.0.5 Linux
Blackdown JDK 1.1.7 Linux
JDK 1.1.8 FreeBSD

Java Platform Operating System Compiler Thread Model
Tower TowerJ 3.1.4 Linux Red Hat Linux 6.0 Intel static many-to-one
IBM JDK 1.1.8 Windows NT Windows NT Workstation 4.0 mixed mode one-to-one
IBM JDK 1.1.8 OS/2 OS/2 Warp Server for e-business mixed mode one-to-one
Microsoft VM 3234 Windows NT Windows NT Workstation 4.0 JIT one-to-one
Microsoft VM 3229 Windows 2000 Windows 2000 Server Release Candidate 2 JIT one-to-one
Appeal JRockit Windows 2000 Windows 2000 Server mixed mode many-to-many
IBM JDK 1.1.8 Linux Red Hat Linux 6.0 Intel JIT one-to-one
Sun HotSpot Server 2.0 Windows NT Windows NT Workstation 4.0 mixed mode one-to-one
Sun JDK 1.3 RC1 Windows NT Windows NT Workstation 4.0 mixed mode one-to-one
Sun JDK 1.2.2-001 Windows NT Windows NT Workstation 4.0 JIT one-to-one
Sun JDK 1.2.1_04 Solaris Solaris 7 Desktop Intel Platform Edition JIT many-to-many
Novell JDK 1.1.7 NetWare NetWare 5 JIT many-to-one
Sun/Inprise JDK 1.2.2 Linux Red Hat Linux 6.0 Intel JIT many-to-one
Blackdown/Inprise JDK 1.2.2 Linux Red Hat Linux 6.0 Intel JIT one-to-one or many-to-one
Blackdown JDK 1.2.2 RC4 Linux Red Hat Linux 6.0 Intel JIT one-to-one or many-to-one
Transvirtual Kaffe 1.0.5 Linux Red Hat Linux 6.0 Intel JIT many-to-one
Blackdown JDK 1.1.7 Linux Red Hat Linux 6.0 Intel none many-to-one
JDK 1.1.8 FreeBSD FreeBSD 3.2-RELEASE none many-to-one
Table 3. The compiler and threading model used by each Java platform.

Compiler

static
A static compiler which takes Java source files or class files and translates them into a native executable or shared library before they run.
mixed mode
A Java VM which translates some of the Java bytecodes into native instructions on the fly while the Java program runs, while letting a Java interpreter handle other parts of the program. Specifically, I use this term to refer to adaptive compilers which use runtime execution profiles to optimize the resulting native instructions.
JIT
A Java VM which translates the Java bytecodes into native instructions on the fly before executing them.
none
A Java VM which uses a normal Java bytecode interpreter.

Thread Model

one-to-one
Each Java thread is mapped one-to-one onto its own operating system thread.
many-to-one
All of the Java threads in the process are mapped onto one operating system process.
many-to-many
More than one Java thread is mapped to one operating system thread, and a Java thread may be mapped to different operating system threads during its lifetime.

Valid HTML 4.0! Valid CSS! Copyright © 2000 Volano LLC. All rights reserved.
Contact John Neffenger with questions or comments.