Hot to solve the “Too many Open Files” error in Java applications

This tutorial will discuss how to fix one of the most common errors for Java applications: “Too many open files“.

The error Java IOException “Too many open files” can happen on high-load servers and it means that a process has opened too many files (file descriptors) and cannot open new ones. In Linux, the maximum open file limits are set by default for each process or user and the defaut values are quite small.

Also note that socket connections are treated like files and they use file descriptor, which is a limited resource.

You can approach this issue with the following checklist:

1) Check what your application is doing. Are you using resources (Sockets/IO Streams/Database connections) without closing properly the connection?

A common way to release safely resources is to use the try-with-resources statement.

The try-with-resources statement is a try statement that declares one or more resources. A resource is an object that must be closed after the program is finished with it. The try-with-resources statement ensures that each resource is closed at the end of the statement. Any object that implements java.lang.AutoCloseable, which includes all objects which implement java.io.Closeable, can be used as a resource.

The following example reads the first line from a file. It uses an instance of BufferedReader to read data from the file. BufferedReader is a resource that must be closed after the program is finished with it:

static String readFromFile(String path) throws IOException {
    try (BufferedReader br =
                   new BufferedReader(new FileReader(path))) {
        return br.readLine();
    }
}

Also, you should consider that, even if you are properly closing your handles, that handle will be disposed only during the Garbage collection phase. It is worth checking, using a tool like Eclipse MAT, which objects are retained in memory when the error “Too many open files” is issued.

2) If you don’t find any apparent flaw in your code, you should check the Global OS limits for your machine.

On a Linux Box the global number of maximum open files is configured in the /proc/sys/file-max file. You can use the sysctl command (with root privileges) to check the current value:

$ sudo /sbin/sysctl fs.file-max
fs.file-max = 3220524

The default value for fs.file-max can change depending on the OS version you are using, however it can be calculated to approximately 1/10 of physical RAM size at boot. If the calculated value is smaller than NR_FILE(8192), the default value is 8192.

Then, investigate on the Java process. Determine the pid of the Java process first:

$ ps -ef | grep jboss

frances+  7790  7671 58 12:10 pts/0    00:01:17 /home/java/jdk-11.0.4/bin/java -D[Standalone] -server -Xlog:gc*:file=/home/jboss/wildfly-23.0.0.Final/standalone/log/gc.log:time,uptimemillis:filecount=5,filesize=3M -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman,org.graalvm.visualvm -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/home/jboss/wildfly-23.0.0.Final/standalone/log/server.log -Dlogging.configuration=file:/home//jboss/wildfly-23.0.0.Final/standalone/configuration/logging.properties -jar /home/jboss/wildfly-23.0.0.Final/jboss-modules.jar -mp /home//jboss/wildfly-23.0.0.Final/modules org.jboss.as.standalone -Djboss.home.dir=/home//jboss/wildfly-23.0.0.Final -Djboss.server.base.dir=/home//jboss/wildfly-23.0.0.Final/standalone -c standalone-ha.xml

Then, check the number of file handles which are opened by the process:

$ lsof -p 7790 | wc -l
811

If you are not sure about which process is hogging your file descriptor table, you can run the following command, which will return the list of file descriptors for each process:

$ lsof | awk '{print $2}' | sort | uniq -c | sort -n

On the other hand, if you want to have the list of handles per user, you can run lsof with the -u parameter:

$ lsof -u jboss | wc -l

So, right now you have the count of open handles on your machine. Besides it, you should also check what are actually doing your sockets. This can be verified with the netstat command. For example, to verify the status of the sockets opened for a single process, you can execute the following command:

$ netstat -tulpn | grep 7790
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:54200         0.0.0.0:*               LISTEN      7790/java           
tcp        0      0 127.0.0.1:8443          0.0.0.0:*               LISTEN      7790/java           
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN      7790/java           
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      7790/java           
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      7790/java           
udp        0      0 224.0.1.105:23364       0.0.0.0:*                           7790/java           
udp        0      0 230.0.0.4:45688         0.0.0.0:*                           7790/java           
udp        0      0 127.0.0.1:55200         0.0.0.0:*                           7790/java           
udp        0      0 230.0.0.4:45688         0.0.0.0:*               TIME_WAIT   7790/java           
udp        0      0 230.0.0.4:45689         0.0.0.0:*               TIME_WAIT   7790/java  

From the netstat output, we can see a small number of TIME_WAIT sockets, which is absolutely normal. You should worry if you detect thousands of active TIME WAIT sockets. If that happens, you should consider some possible actions such as:

  • Make sure you close the TCP connection on the client before closing it on the server.
  • Consider reducing the timeout of TIME_WAIT sockets. In most Linux machines, you can do it by adding the following contents to the sysctl.conf file (f.e.reduce to 30 seconds):
net.ipv4.tcp_syncookies = 1 
net.ipv4.tcp_tw_reuse = 1 
net.ipv4.tcp_tw_recycle = 1 
net.ipv4.tcp_timestamps = 1 
net.ipv4.tcp_fin_timeout = 30 
net.nf_conntrack_max = 655360 
net.netfilter.nf_conntrack_tcp_timeout_established  = 1200  
  • Use more client ports by setting net.ipv4.ip_local_port_range to a wider range.
  • Have the application to Listen for more server ports. (Example httpd defaults to port 80, you can add extra ports).
  • Add more client IPs by configuring additional IP on the load balancer and use them in a round-robin fashion.

Also, consider that other processes are running on your machine and you should account for them as well, if your machine is not dedicated only to your Java application.

That being said, what to do if the the number of process handles is greater than the fs.file-max? you can increase the value for fs.file-max as follows:

$ sudo/sbin/sysctl -w fs.file-max=<NEWVALUE>

3) Check the limits imposed per user/shell

On a Linux RHEL/Fedora, these limits are configured through the Pluggable Authentication Module (PAM). These limits can be configured in the file /etc/security/limits.conf. To verify the current limits for your user, run the command ulimit:

$ ulimit -n

1024

To change this value for the user jboss, who is running the Java application, change as follows the /etc/security/limits.conf file:

jboss soft nofile 2048
jboss hard nofile 2048

You need to login again and restart the process for the changes to take effect.

Finally please note that the user limits will be ignored when your application is started in a cron job as the cron does not uses a login shell.