Home → Programming → Linux 2.2 Sockets and Threads
By John Neffenger
March 1, 2000
This document describes how to modify the per-process file descriptor limit and task limit of the Linux version 2.2 kernel to increase the number of sockets and threads that a Java server application can create. In particular, it describes the socket and thread requirements of the VOLANO® chat server, but similar requirements are likely to exist in any server application written for the Java platform. Note that the current Linux kernel version is 2.6, so this information is unlikely to apply to any recent Linux distribution.
When running on the Linux 2.2 kernel, each process is limited by default to 1,024 file descriptors. Since the VolanoChat server requires one file descriptor for each person chatting through a VolanoChat applet, you are limited by default to a maximum of 1,024 simultaneous users when using the Linux 2.2 kernel.
Some Java virtual machines, such as the IBM Developer Kit for Linux, Java Technology Edition, use native threads on Linux which require one task for each Java thread. Since VolanoChat requires two Java threads for each connection, you may need to increase the maximum number of tasks per user for such virtual machines. The default maximum number of tasks in a Linux 2.2 system is 512, with a maximum of 256 tasks per user. So by default, a VolanoChat server running under IBM JDK 1.1.6 for Linux can support a maximum of 128 people chatting at the same time.
You can increase the limits testing the the number of file descriptors and the number of tasks in the Linux 2.2 kernel by following the steps below. These steps were performed on a Red Hat Linux 6.0 system using Linux kernel 2.2.5-15. The instructions may be different for your Linux system.
To increase the maximum file descriptors per process to 4,096 (from 1,024) and the maximum tasks per user to 4,090 (from 256) on Linux 2.2, do the following steps.
Change the following C header files.
In /usr/include/linux/tasks.h, change:
NR_TASKS 512 to 4090 MAX_TASKS_PER_USER (NR_TASKS/2) to NR_TASKS
In /usr/include/linux/limits.h, change:
NR_OPEN 1024 to 4096 OPEN_MAX 256 to 4096
In /usr/include/linux/posix_types.h, change:
__FD_SETSIZE 1024 to 4096
In /usr/include/bits/types.h, change:
__FD_SETSIZE 1024 to 4096
To allow users to increase their file descriptor limits, change the following configuration files:
In /etc/security/limits.conf, add the lines:
* soft nofile 1024 * hard nofile 4096
In /etc/pam.d/login, add:
session required /lib/security/pam_limits.so
/etc/rc.d/rc.local startup script:
# Increase system-wide file descriptor limit. echo 8192 > /proc/sys/fs/file-max echo 24576 > /proc/sys/fs/inode-max
Change to the directory /usr/src/linux and set up the kernel sources with:
make mrproper
Make any other kernel configuration changes, such as setting the processor family type, by entering the command below. Save your changes and exit the configuration panel when you are done.
make xconfig
Prepare for the kernel build by entering:
make dep make clean
Rebuild your kernel with the commands:
make bzImage make modules
Backup your old modules directory by entering (for example):
mv /lib/modules/2.2.5-15 /lib/modules/2.2.5-15.old
Create a new set of kernel modules with the command:
make modules_install
Backup your old kernel by entering (for example):
mv /boot/vmlinuz /boot/vmlinuz.old
Copy over your new kernel image with:
cp /usr/src/linux/arch/i386/boot/bzImage /boot/vmlinuz
Edit the Linux loader configuration in /etc/lilo.conf so that your new kernel is the default and your old kernel is still available from the boot prompt, as in the example shown below:
boot=/dev/hda
map=/boot/map
install=/boot/boot.b
prompt
timeout=50
image=/boot/vmlinuz
label=linux
root=/dev/hda1
read-only
image=/boot/vmlinuz.old
label=linux.old
root=/dev/hda1
read-only
Generate the new Linux loader configuration by running:
/sbin/lilo
Shutdown and reboot with your new kernel by entering:
shutdown -r now
After rebooting, verify that the changes were effective as shown in the sections below.
Even after you rebuild your Linux 2.2 kernel to increase the maximum number of threads per Linux user from 256 to 4,090, there are still definitions in the thread library itself which limit the number of threads per process to less than 1,000. The VolanoChat server requires two threads for each person chatting through a VolanoChat applet, so the LinuxThreads implementation creates a limit of 500 people to a single VolanoChat server.
You can increase this limit to 8,192 threads with the following steps. Even though the thread library will allow 8,192 threads per process after these changes, you'll still be limited by the Linux 2.2 kernel to a maximum of 4,090 threads for the Linux user account running the VolanoChat server after the changes in the previous section. The steps below are shown for a Red Hat Linux 6.0 system, so they may be different for your Linux distribution.
Install the Glibc 2.1.1 source package from /mnt/cdrom/SRPMS/glibc-2.1.1-6.src.rpm on the Red Hat Linux CD-ROM #2.
Unpack the source archive /usr/src/redhat/SOURCES/glibc-990416.tar.gz so that you can modify two of the files.
Change the following C header files.
In /usr/src/redhat/SOURCES/glibc/linuxthreads/internals.h, change the size of the thread stack reserve from 2 megabytes down to 256 kilobytes (with a page size of 4,096 bytes):
STACK_SIZE (2 * 1024 * 1024) -> (64 * PAGE_SIZE)
In sysdeps/unix/sysv/linux/bits/local_lim.h under /usr/src/redhat/SOURCES/glibc/linuxthreads, change the Posix thread implementation limit from 1,024 per process to 8,192 per process:
PTHREAD_THREADS_MAX 1024 -> 8192
Rebuild the source archive, /usr/src/redhat/SOURCES/glibc-990416.tar.gz, to include your changes, replacing the original copy.
Change to the RPM specification directory, /usr/src/redhat/SPECS, and enter the command:
rpm -ba glibc-2.1.spec
After a few hours, the new LinuxThreads library will be found as libpthread.so under:
/usr/src/redhat/BUILD/glibc/build-i386-linux/linuxthread
Copy the new thread library into your own lib subdirectory:
cd /usr/src/redhat/BUILD/glibc/build-i386-linux cp linuxthread/libpthread.so ~/lib cd ~/lib ln -s libpthread.so libpthread.so.0
Define the environment variable LD_LIBRARY_PATH to include the subdirectory containing the new thread library libpthread.so.
Verify the new thread limits as instructed below.
Increase your new limits to their maximum by entering "ulimit -n 4096" for the Bash shell or "unlimit" for the C shell.
You can then check whether your changes were effective by running the following C program:
#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>
void main(void) {
int k = 3;
while (dup(0) != -1)
k++;
printf("Opens = %d Fdsize = %d\n", k, sizeof(fd_set) * 8);
}
Both numbers printed should give your new file descriptor limit.
You can use the following program to verify your new thread limits in the modified Linux 2.2 kernel and in the modified LinuxThreads library.
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#define MAX_THREADS 10000
int i;
void run(void) {
char c;
if (i < 10)
printf("Address of c = %u KB\n", (unsigned int) &c / 1024);
sleep(60 * 60);
}
int main(int argc, char *argv[]) {
int rc = 0;
pthread_t thread[MAX_THREADS];
printf("Creating threads ...\n");
for (i = 0; i < MAX_THREADS && rc == 0; i++) {
rc = pthread_create(&(thread[i]), NULL, (void *) &run, NULL);
if (rc == 0) {
pthread_detach(thread[i]);
if ((i + 1) % 1000 == 0)
printf("%i threads so far ...\n", i + 1);
}
else
printf("Failed with return code %i creating thread %i.\n",
rc, i + 1);
}
exit(0);
}
Compile the program with:
gcc -lpthread -o count count.c
Running with the original LinuxThreads library will print:
blue:/home/john> count Creating threads ... Address of c = 3137535 KB Address of c = 3135487 KB Address of c = 3133439 KB Address of c = 3131391 KB Address of c = 3129343 KB Address of c = 3127295 KB Address of c = 3125247 KB Address of c = 3123199 KB Address of c = 3121151 KB 1000 threads so far ... Failed with return code 11 creating thread 1023.
while running with the modified LinuxThreads library will print:
blue:/home/john> count Creating threads ... Address of c = 3144703 KB Address of c = 3144447 KB Address of c = 3144191 KB Address of c = 3143935 KB Address of c = 3143679 KB Address of c = 3143423 KB Address of c = 3143167 KB Address of c = 3142911 KB Address of c = 3142655 KB 1000 threads so far ... 2000 threads so far ... 3000 threads so far ... 4000 threads so far ...
You may need to enter Ctrl-C to exit the program.
Notice that the thread stack addresses are spaced apart by 256 kilobytes instead of the earlier reserve of 2 megabytes per stack. Also notice that the program can create threads right up to the Linux 2.2 kernel limit of 4,090 for the system rather than the earlier LinuxThreads limit of 1,024 per process.