The tests presented in this report look at the performance and network scalability of 17 Java virtual machines on 7 operating systems using a common Intel hardware platform running VolanoMark 2.1. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:
When run at high connection counts, VolanoMark essentially measures the consequences of omitting non-blocking read and write operations from the Java platform, as summarized by this Java Developer Connection request for enhancement (free registration required):
Note that I have been unsuccessful running VolanoMark on the following Java platforms:
For background information and download locations, please see the Java Benchmarks page on Volano's Web site. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments, along with a table showing the compiler and threading model of each Java platform.
Based on our experience and these test results, we recommend the following Java platforms for running a pure Java server application:
The performance test was executed with the following commands on a local loop-back connection, using the heap size options shown below where possible:
java -ms8m -mx64m COM.volano.Main
java -ms8m -mx64m COM.volano.Mark -count 100
The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.
Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.
Java Platform | Score |
---|---|
Tower TowerJ 3.1.4 Linux (a) IBM JDK 1.1.8 Windows NT IBM JDK 1.1.8 OS/2 Microsoft VM 3229 Windows 2000 IBM JDK 1.1.8 Linux Sun HotSpot 1.0.1 Windows NT Sun JDK 1.3 Beta Windows NT Sun JDK 1.2.2 Windows NT Microsoft VM 3186 Windows NT Sun JDK 1.2.1_04 Solaris Novell JDK 1.1.7 NetWare (b) Sun JDK 1.2.1_03 Solaris Inprise JDK 1.2.2 RC1 Linux Blackdown JDK 1.2.2 RC3 Linux Transvirtual Kaffe 1.0.5 Linux (c) Blackdown JDK 1.1.7 Linux JDK 1.1.8 FreeBSD |
2309 2279 2136 1970 1776 1601 1550 1485 1478 1358 1229 1051 1039 943 908 285 173 |
Java Platform | Operating System | Results | Score |
---|---|---|---|
Tower TowerJ 3.1.4 Linux (a) | Red Hat Linux 6.0 Intel | 2303, 2313, 2313, 2300 | 2309 |
IBM JDK 1.1.8 Windows NT | Windows NT Workstation 4.0 | 2282, 2276, 2280, 2282 | 2279 |
IBM JDK 1.1.8 OS/2 | OS/2 Warp Server for e-business | 2145, 2141, 2118, 2150 | 2136 |
Microsoft VM 3229 Windows 2000 | Windows 2000 Server Release Candidate 2 | 1948, 1965, 1976, 1968 | 1970 |
IBM JDK 1.1.8 Linux | Red Hat Linux 6.0 Intel | 1227, 1785, 1770, 1773 | 1776 |
Sun HotSpot 1.0.1 Windows NT | Windows NT Workstation 4.0 | 1558, 1611, 1604, 1589 | 1601 |
Sun JDK 1.3 Beta Windows NT | Windows NT Workstation 4.0 | 1538, 1550, 1556, 1545 | 1550 |
Sun JDK 1.2.2 Windows NT | Windows NT Workstation 4.0 | 1496, 1481, 1489, 1485 | 1485 |
Microsoft VM 3186 Windows NT | Windows NT Workstation 4.0 | 1469, 1484, 1478, 1473 | 1478 |
Sun JDK 1.2.1_04 Solaris | Solaris 7 Desktop Intel Platform Edition | 1371, 1344, 1374, 1355 | 1358 |
Novell JDK 1.1.7 NetWare (b) | NetWare 5 | 1227, 1232, 1229, 1226 | 1229 |
Sun JDK 1.2.1_03 Solaris | Solaris 7 Desktop Intel Platform Edition | 1118, 1055, 1050, 1049 | 1051 |
Inprise JDK 1.2.2 RC1 Linux | Red Hat Linux 6.0 Intel | 1032, 1040, 1036, 1040 | 1039 |
Blackdown JDK 1.2.2 RC3 Linux | Red Hat Linux 6.1 Intel | 939, 945, 942, 943 | 943 |
Transvirtual Kaffe 1.0.5 Linux (c) | Red Hat Linux 6.0 Intel | 906, 905, 909, 909 | 908 |
Blackdown JDK 1.1.7 Linux | Red Hat Linux 6.0 Intel | 284, 285, 286, 283 | 285 |
JDK 1.1.8 FreeBSD | FreeBSD 3.2-RELEASE | 169, 173, 173, 172 | 173 |
The network scalability test was executed with the following commands over an isolated 10-Mbps Ethernet connection, using the heap and stack size options shown below where possible:
java -ms8m -mx128m -ss32k COM.volano.Main
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150
java -ms8m -mx128m -ss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200
See the COM.volano.Mark command synopsis for a complete description of all options.
The client test driver was executed under the Sun JDK 1.2.1_03 Production Release for Solaris using Solaris 7 on a dual 200-MHz Sun UltraSPARC with 256 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.
Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Environments section for details on the hardware platform, operating systems, and Java virtual machine environments.
1000 | 2000 | 3000 | 4000 | |||
---|---|---|---|---|---|---|
|
Java Platform | 1000 | 2000 | 3000 | 4000 | Notes |
---|---|---|---|---|---|
Tower TowerJ 3.1.4 Linux | 2486 | 1357 | 739 | 421 | No errors! |
IBM JDK 1.1.8 Windows NT | 2623 | 1826 | 1081 | ---- | "java.net.SocketException: Socket read failed: 10107" at 3553/4000 connections. |
IBM JDK 1.1.8 OS/2 | 1521 | ---- | ---- | ---- | "java.lang.OutOfMemoryError" at 1098/2000 connections. |
Microsoft VM 3229 Windows 2000 | 1408 | 708 | 406 | ---- | "java.lang.OutOfMemoryError" at 3785/4000 connections. |
IBM JDK 1.1.8 Linux | ---- | ---- | ---- | ---- | Process hangs in hard run at 500/1000 connections and dumps core when killed. |
Sun HotSpot 1.0.1 Windows NT | ---- | ---- | ---- | ---- | "java.lang.OutOfMemoryError" at 902/1000 connections. |
Sun JDK 1.3 Beta Windows NT | 1474 | 856 | 523 | ---- | "java.lang.OutOfMemoryError" at 3272/4000 connections. |
Sun JDK 1.2.2 Windows NT | ---- | ---- | ---- | ---- | "java.lang.OutOfMemoryError" at 926/1000 connections. |
Microsoft VM 3186 Windows NT | 1468 | 770 | ---- | ---- | "java.net.SocketException: No available buffer space" on 3000-connection test. |
Sun JDK 1.2.1_04 Solaris | 1875 | 1040 | 659 | ---- | "Bus error - core dumped" on 4000-connection test. |
Novell JDK 1.1.7 NetWare | 1575 | ---- | ---- | ---- | Process hangs with disk thrashing at 1960/2000 connections. |
Sun JDK 1.2.1_03 Solaris | 1794 | 1058 | 613 | 354 | No errors! |
Inprise JDK 1.2.2 RC1 Linux | ---- | ---- | ---- | ---- | Fails with "java.net.SocketException: Interrupted system call". |
Blackdown JDK 1.2.2 RC3 Linux | 1153 | ---- | ---- | ---- | Fails the 2000-connection test with "SIGSEGV 11* segmentation violation" at just over 1000 connections. |
Transvirtual Kaffe 1.0.5 Linux | ---- | ---- | ---- | ---- | Hangs with zero percent CPU usage after starting, even at just 500 connections. Dumps core when the client side is killed to stop the test. |
Blackdown JDK 1.1.7 Linux | 165 | ---- | ---- | ---- | Dumps core at just over 1000 connections. |
JDK 1.1.8 FreeBSD | 37 | ---- | ---- | ---- | Client begins failing at 1017 connections with "java.net.NoRouteToHostException: Connection timed out". |
All tests ran identical copies of VolanoMark 2.1.2 on identical hardware -- a 200-MHz Intel Pentium Pro processor with a 256-kilobyte L2 cache and 256 megabytes of RAM on an Intel VS440FX motherboard running American Megatrends AMIBIOS Version 1.00.18.CS1 dated "08/28/98-13:59". The network card is a 3Com Etherlink III PCI Bus-Master Adapter Model 3C590.
"version 3.1.4.0 x86-linux"
TowerJ_3_1_4_0_x86_linux.class
(12,913,370 bytes).
-Dtowerj.poll
for the
network scalability test.
Main.tj
and
Mark.tj
.
"JDK 1.1.8 IBM build n118p-19990728 (JIT enabled: ibmjitc V3.5-IBMJDK1.1-19990728)"
ibm-jdk-n118p-win32-x86.zip
(10,810,316 bytes).
ibmjitc
just-in-time compiler.
14.039F_UNI
)
"JDK 1.1.8 IBM build o118-19990728 (JIT enabled: javax V3.5-IBMJDK1.1-19990728)"
javainuf.exe
(20,805,936 bytes) and
javaintk.exe
(15,273,452 bytes).
javax
just-in-time compiler.
CONFIG.SYS
file to increase the thread
limit from 1024 to 4095 (THREADS=4095)
and the initial swap
file size from 2 MB to 32 MB
(SWAPPATH=C:\OS2\SYSTEM 2048 32768)
.
"1.1.4"
5.00.3229
"JDK 1.1.8 IBM build l118-19991013 (JIT enabled: jitc)"
ibm-jdk-l118-linux-x86.tgz
(10,555,308 bytes).
jitc
just-in-time compiler.
HotSpot VM (1.0.1, mixed mode, build g)
.
hotspot1_0_1-win.zip
(1,256,720 bytes).
"X"
, and the stack options are not available.
"1.3beta, Java(TM) 2 Runtime Environment,
Standard Edition (build 1.3beta-0), Java(TM) HotSpot Client VM (build
1.3beta-0, mixed mode)"
jdk1_3beta-win.exe
(25,662,367 bytes).
"X"
, and the stack options are not available.
"Classic VM (build JDK-1.2.2-W, native threads, symcjit)"
jdk1_2_2-win.exe
(20,422,291 bytes).
symcjit
Symantec
Just-in-Time compiler.
"X"
, and the stack options are not available.
"1.1.4"
5.00.3186
msjavx86.exe
(6,632,392 bytes).
"Solaris_JDK_1.2.1_04_pre-release"
Solaris_JDK_1.2.1_04_pre-release_i386.bin
(18,607,361
bytes).
sunwjit
just-in-time compiler.
"X"
.
rlim_fd_max
variable in /etc/system
.
"1.1.7 B"
jvm.exe
(18,015,477 bytes).
symcjit
just-in-time compiler
(Symantec Java! JustInTime Compiler Version 3.10.106(x) for JDK 1.1.x).
java.cfg
file to enable the just-in-time
compiler (JAVA_COMPILER=symcjit)
.
"Solaris_JDK_1.2.1_03"
1.2.1_03_jdk_i386.tar.Z
(26,264,747
bytes).
sunwjit
just-in-time compiler.
"X"
.
rlim_fd_max
variable in /etc/system
.
"1.2.2", "Classic VM (build 1.2.2-I, green threads, javacomp)"
jdk1_2_2rc1-linux-i386.tar.gz
(21,794,070 bytes).
javacomp
just-in-time compiler.
"X"
.
"1.2.2", "Classic VM (build Linux_JDK_RC3, green
threads, sunwjit)"
jdk-1.2.2-RC3-linux-i386.sh
(21,028,547
bytes).
sunwjit
just-in-time
compiler.
"-green"
option for user-level threads.
"X"
.
"Engine: Just-in-time v3 Version: 1.0.5 Java Version: 1.1"
kaffe-1.0.5.tar.gz
(3,145,844 bytes).
kaffe.jit
just-in-time compiler.
"Linux_JDK_1.1.7B_v3_green_threads"
jdk_1.1.7-v3-glibc-x86.tar.bz2
(10,130,082 bytes).
"jdk1.1.8-FreeBSD:1999/7/19"
jdk1.1.8_ELF.V99-7-19.tar.gz
(11,337,773 bytes).
sysctl
to modify the kern.maxfilesperproc
and
kern.maxfiles
variables in /etc/rc.local
.
Java Platform | Operating System | Compiler | Thread Model |
---|---|---|---|
Tower TowerJ 3.1.4 Linux | Red Hat Linux 6.0 Intel | static | many-to-one |
IBM JDK 1.1.8 Windows NT | Windows NT Workstation 4.0 | mixed mode | one-to-one |
IBM JDK 1.1.8 OS/2 | OS/2 Warp Server for e-business | mixed mode | one-to-one |
Microsoft VM 3229 Windows 2000 | Windows 2000 Server Release Candidate 2 | JIT | one-to-one |
IBM JDK 1.1.8 Linux | Red Hat Linux 6.0 Intel | JIT | one-to-one |
Sun HotSpot 1.0.1 Windows NT | Windows NT Workstation 4.0 | mixed mode | one-to-one |
Sun JDK 1.3 Beta Windows NT | Windows NT Workstation 4.0 | mixed mode | one-to-one |
Sun JDK 1.2.2 Windows NT | Windows NT Workstation 4.0 | JIT | one-to-one |
Microsoft VM 3186 Windows NT | Windows NT Workstation 4.0 | JIT | one-to-one |
Sun JDK 1.2.1_04 Solaris | Solaris 7 Desktop Intel Platform Edition | JIT | many-to-many |
Novell JDK 1.1.7 NetWare | NetWare 5 | JIT | many-to-one |
Sun JDK 1.2.1_03 Solaris | Solaris 7 Desktop Intel Platform Edition | JIT | many-to-many |
Inprise JDK 1.2.2 RC1 Linux | Red Hat Linux 6.0 Intel | JIT | many-to-one |
Blackdown JDK 1.2.2 RC3 Linux | Red Hat Linux 6.1 Intel | JIT | one-to-one or many-to-one |
Transvirtual Kaffe 1.0.5 Linux | Red Hat Linux 6.0 Intel | JIT | many-to-one |
Blackdown JDK 1.1.7 Linux | Red Hat Linux 6.0 Intel | none | many-to-one |
JDK 1.1.8 FreeBSD | FreeBSD 3.2-RELEASE | none | many-to-one |