For background information and download locations, please see the Java Benchmarks page on Volano's Web site. Since the server side of the VolanoMark™ benchmark is our VolanoChat server, these test results help our customers choose the best Java virtual machine for their environment.
VolanoMark is a pure Java server benchmark characterized by long-lasting network connections and high thread counts. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second.
Bug Id 4075058, "java.io: Add support for non-blocking I/O," explains why you need lots of threads in order to handle dedicated connections to a pure Java server. JSR 51, "New I/O APIs for the Java Platform," explains how Sun plans to change that.
Well, just about everything is new -- the network, the hardware, the operating systems, and the Java virtual machines. This time I restricted testing to the Java 2 Standard Edition and excluded IBM OS/2 Warp Server and Novell NetWare, but that still left 12 Java platforms on 4 operating systems.
This is the first time that Tower's TowerJ did not come out on top in the performance test. TowerJ's problems in the network scalability test, though, are most likely due to my running the test on Red Hat Linux 6.2, which Tower has not yet tested. In the network scalability test, only Appeal's JRockit 1.1.2 on Windows and Blackdown's J2SE 1.2.2-FCS on Linux made it all the way to 5,000 concurrent connections.
On Linux, the latest "FCS" version of Blackdown's port solved problems that I hit trying to run Blackdown's earlier "RC4" version on Red Hat 6.2 (Linux 2.2.14, Glibc 2.1.3), even though "RC4" ran fine on Red Hat 6.0 (Linux 2.2.5, Glibc 2.1.1). In these tests and in the previous Volano Report back in March, the Blackdown releases have been consistently more stable at high connection counts than Sun's own releases of the same version.
On Solaris Intel, Sun's older ExactVM (1.2.2) outperformed the new HotSpot Server VM (1.3.0), although results may be quite different for Solaris SPARC. Both virtual machines do very well in the network scalability test, failing only when there are more than 4,000 concurrent connection. Our customers have tested successfully up to 10,000 concurrent connections with Java on Solaris SPARC, so these problems may be limited to Java on the Solaris Intel Edition.
Without a native binary release of Java 2 Standard Edition for FreeBSD, I tested the Linux Java VMs with FreeBSD's Linux emulation package. The earlier Blackdown "RC4" release and Sun's 1.3.0 Classic VM worked quite well up to about 980 connections on FreeBSD, but the latest "FCS" Blackdown release failed to run at all. In addition, neither Sun's nor Blackdown's native thread Java VMs, including HotSpot, worked under the FreeBSD Linux emulation.
All of the Windows Java virtual machines did well, but only Appeal's JRockit gets around the Windows stack-based thread limit by mapping the Java threads onto its own smaller set of native threads.
The tests presented in this report look at the performance and network scalability of 12 Java virtual machines on 4 operating systems using a common Intel hardware platform running VolanoMark 2.1. You can find links to VolanoMark test results on other (non-Intel) hardware platforms on our Java Benchmarks page. VolanoMark attempts to answer two questions about Java virtual machines used in network server environments -- especially those environments involving a large number of dedicated socket connections:
See the Details section for details on the hardware platform, operating systems, and Java virtual machine environments.
The performance test was executed with the following commands on a local loopback connection, using the heap size options shown below where possible:
java -Xms8m -Xmx64m COM.volano.Mainjava -Xms8m -Xmx64m COM.volano.Mark -count 100See the COM.volano.Mark command synopsis for a complete description of all options. The operating system was rebooted before each set of tests for a particular Java virtual machine. The server side was not restarted before each run of the client benchmark. The client benchmark ran four times, with the final score as the average of the last three results.
Scores are the throughput of the server in messages per second. Bigger numbers are faster. See the Details section for details on the hardware platform, operating systems, and Java virtual machine environments.
Table 1. VolanoMark 2.1.2 performance test, measuring throughput in messages per second with 200 concurrent local loopback connections.
The network scalability test was executed with the following commands over a 100-Mbps Ethernet cross-over cable connection, using the heap and stack size options shown below where possible:
java -Xms8m -Xmx256m -Xss32k COM.volano.Mainjava -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 50java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 100java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 150java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 200java -Xms8m -Xmx256m -Xss32k COM.volano.Mark -host xxx.yyy.zzz -rooms 250See the COM.volano.Mark command synopsis for a complete description of all options. The client test driver was executed under the Sun J2SE 1.3.0 Release Candidate HotSpot Server VM using Solaris 8 (6/00) on a 500-MHz Intel Pentium III with 384 megabytes of RAM. The operating systems on both sides were rebooted before each set of tests for a particular Java virtual machine. The VolanoMark server was not restarted between client test runs.
Scores are the throughput of the server in messages per second based on the total number of concurrent connections. Bigger numbers are faster. See the Details section for details on the hardware platform, operating systems, and Java virtual machine environments.
Table 2. VolanoMark 2.1.2 network scalability test, measuring throughput in messages per second with 1000 through 5000 concurrent network connections.
All tests ran identical copies of VolanoMark 2.1.2 on identical hardware.
The server machine under test is a Dell OptiPlex GXpro with a 200-MHz Intel Pentium Pro processor, 256 kilobytes L2 cache, 320 megabytes ECC EDO RAM, a 3Com EtherLink XL 10/100 PCI TX NIC (3C905B-TX), and Phoenix ROM BIOS PLUS Version 1.10 A10.
The client side driving the network tests is a Dell OptiPlex GX1 with a 500-MHz Intel Pentium III processor, 512 kilobytes L2 cache, 384 megabytes ECC SDRAM, a 3Com EtherLink XL 10/100 PCI TX NIC (3C905B-TX), and Dell OptiPlex GX1 500L+ BIOS Revision A08.
The following Unix shell scripts and Windows batch files were used for the loopback performance tests with 200 concurrent connections.
The server side used:
The client side used:
For a sleep function, the Windows batch files used sleep.java.
The following Unix shell scripts and Windows batch files were used for the network scalability tests with 1000 through 5000 concurrent connections.
The server side used:
The client side used:
Operating system version:
FreeBSD ppro.sf.volano.net 4.0-RELEASE FreeBSD 4.0-RELEASE #0: Mon Mar 20 22:50:22 GMT 2000 root@monster.cdrom.com:/usr/src/sys/compile/GENERIC i386
Contents of /etc/sysctl.conf:
# Increase file descriptor limits from their defaults of 1064. kern.maxfiles=16384 # System-wide limit kern.maxfilesperproc=8192 # Per-process limit
Downloaded from:
http://www.blackdown.org/java-linux/ports.html
Ran using the Linux emulation package (Linux 2.2.12, Glibc 2.1.2) and the
Inprise just-in-time compiler.
Changed the calls to expr in
jre1.2.2/bin/.java_wrapper as follows:
expr -> /usr/compat/linux/usr/bin/expr
Java version:
java version "1.2.2" Classic VM (build Linux_JDK_1.2.2_FCS, green threads, javacomp)
Downloaded from:
http://java.sun.com/products/jdk/1.3/
Ran using the Linux emulation package (Linux 2.2.12, Glibc 2.1.2) and the
Inprise just-in-time compiler.
Changed the calls to /usr/bin/expr in
jre1.3/bin/.java_wrapper as follows:
/usr/bin/expr -> /usr/compat/linux/usr/bin/expr
Java version:
java version "1.3.0beta_refresh" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09) Classic VM (build 1.3.0beta_refresh-b09, green threads, javacomp)
Operating system version:
Microsoft Windows 2000 Professional Version 5.0.2195 Service Pack 1 Build 2195
Optimized performance for background services (rather than applications) under Start, Settings, Control Panel, System, Advanced, Performance Options, Application response. This change specifies that all programs receive equal amounts of processor resources.
Downloaded from:
http://www.jrockit.com/
Windows 2000 Service Pack 1 broke the stack size options for JRockit,
so neither the Xss32k option nor the
Xexpandstacks option were used.
Java version:
java version "1.3" JRockit(tm) Virtual Machine (version 1.1.2, build sarek130, Thin Threads) Java Home: C:\Program Files\JavaSoft\JRE\1.3 Using bootclasspath C:\Program Files\JavaSoft\JRE\1.3\lib\rt.jar; C:\Program Files\JavaSoft\JRE\1.3\lib\i18n.jar; C:\Program Files\JavaSoft\JRE\1.3\classes
Downloaded from:
http://www.ibm.com/java/jdk/download/index.html
Java version:
java version "1.3.0" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0) Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cn130-20000622 (JIT enabled: jitc))
Downloaded from:
http://java.sun.com/products/jdk/1.3/
Java version:
java version "1.3.0" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-C) Java HotSpot(TM) Server VM (build 2.0fcs-E, mixed mode)
Operating system version (Linux 2.2.14, Glibc 2.1.3):
Red Hat Linux release 6.2 (Zoot) Linux ppro.sf.volano.net 2.2.14-5.0 #1 Tue Mar 7 21:07:39 EST 2000 i686 unknown
Added to /etc/rc.d/rc.local:
# Increase system-wide file descriptor limit. echo 32768 > /proc/sys/fs/file-max echo 131072 > /proc/sys/fs/inode-max
Added to /etc/pam.d/login:
session required /lib/security/pam_limits.so
Added to /etc/security/limits.conf:
* soft nofile 1024 * hard nofile 8192
Downloaded from:
http://www.blackdown.org/java-linux/ports.html
Ran using the Inprise just-in-time compiler.
Java version:
java version "1.2.2" Classic VM (build Linux_JDK_1.2.2_FCS, green threads, javacomp)
Downloaded from:
http://www.ibm.com/java/jdk/download/index.html
Java version:
java version "1.3.0" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0) Classic VM (build 1.3.0, J2RE 1.3.0 IBM build cx130-20000623 (JIT enabled: jitc))
Downloaded from:
http://java.sun.com/products/jdk/1.3/
Java version:
java version "1.3.0beta_refresh" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09) Java HotSpot(TM) Server VM (build 1.3.0beta-b07, mixed mode)
Downloaded from:
http://java.sun.com/products/jdk/1.3/
Ran using the Inprise just-in-time compiler.
Java version:
java version "1.3.0beta_refresh" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0beta_refresh-b09) Classic VM (build 1.3.0beta_refresh-b09, green threads, javacomp)
Downloaded from:
http://www.towerj.com/
Used the TowerJ project files
Main.tj and
Mark.tj.
Built against Blackdown JDK 1.2.2 RC4 with the environment variables:
export TOWERJ=/usr/local/TowerJ export TOWERJ_JAVA_HOME=/usr/local/jdk1.2.2/jre export LD_LIBRARY_PATH=$TOWERJ/lib/x86-linux
Java version:
TowerJ Compiler (version 3.5.3.0 x86-linux) Copyright (c) 2000 Tower Technology Corporation. All rights reserved.
Operating system version:
Solaris 8 6/00 s28x_u1wos_08 INTEL SunOS ppro 5.8 Generic_108529-01 i86pc i386 i86pc
Added patch 108941-07, "Motif 2.1.1_x86: Runtime library patch for Solaris 8_x86," as required by Sun J2SE 1.3.0-RC.
Added to /etc/system:
set rlim_fd_max = 8192 set rlim_fd_cur = 1024
Downloaded from:
http://www.sun.com/software/solaris/java/download.html
Java version:
java version "1.2.2" Solaris VM (build Solaris_JDK_1.2.2_05a, native threads, sunwjit)
Downloaded from:
http://www.sun.com/software/solaris/java/download.html
Java version:
java version "1.3.0" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0-RC) Java HotSpot(TM) Server VM (build 1.3.0-RC, mixed mode)
Copyright © 2000 Volano LLC. All rights reserved.