Scenario: Your application log displays the error message “Too many Open Files“. As a result, application requests are failing and you need to restart the application. In some cases, you also need to reboot your machine. In this article we will discuss File handling best practices and how to prevent File leaks in Java with proper resource management.
Facts: The JVM console (or log file) contains the following error:
java.io.FileNotFoundException: filename (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:106) at java.io.FileInputStream.<init>(FileInputStream.java:66) at java.io.FileReader.<init>(FileReader.java:41)
Or:
java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
What is the solution?
The error Java IOException “Too many open files” can happen on high-load servers and it means that a process has opened too many files (file descriptors) and cannot open new ones. In Linux, the maximum open file limits are set by default for each process or user and the defaut values are quite small.
Also note that socket connections are treated like files and they use file descriptor, which is a limited resource.
You can approach this issue with the following check-list:
- Check the JVM File handles
- Check other processes’ File Handles
- Inspect the limits of your OS
- Inspect user limits
Monitoring the JVM File Handles
Firstly, we need to determine if the Root cause of “Too many open Files” is the JVM process itself.
On a Linux machine, everything is a file: this includes, for example, regular files or sockets. To check the open files per process you can use the Linux command lsof -p <PID> . For example, if your JVM process has the pid 1234:
lsofpid -p 20583
This command will return in the last column the file descriptor name:
java 20583 user mem REG 253,0 395156 785358 /home/application/fileleak1.txt java 20583 user 48r REG 253,0 395156 785358 /home/application/fileleak2.txt
To get just the total number of File Descriptors which are open, just can simply pipe it with the wc command as follows:
On a Windows machine, you can check the open files through the Resource Monitor Tool which contains this information in the Disk section:
If you are seeing too many application file handles, it is likely that you have a leak in your application code. To avoid resource leaks in your code, we recommend to use the try-with-resources statement.
The try-with-resources statement is a try statement that declares one or more resources. A resource is an object that must be closed after the program is finished with it. The try-with-resources statement ensures an automatic close at the end of the statement. You can use this construct with any object implementing java.io.Closeable interface.
The following example shows a File Handling best practice which reads the first line from a file using an instance of BufferedReader:
static String readFromFile(String path) throws IOException { try (BufferedReader br = new BufferedReader(new FileReader(path))) { return br.readLine(); } }
To learn more about try-with-resources, check this article Using try-with-resources to close database connections
Finally, consider that, even though you are properly closing your handles, it will be actually disposed after the Garbage collection phase. It is worth checking, using a tool like Eclipse MAT, which are the top objects in memory when you spot the error “Too many open files”.
Monitor other processes’ File Handles
At this point, we have determined that your application does not have File Leaks. We need to extend the search to other processes running on your Machine.
The following command returns the list of file descriptors for each process:
$ lsof | awk '{print $2}' | sort | uniq -c | sort -n
On the other hand, if you want to have the list of handles per user, you can run lsof with the -u parameter:
$ lsof -u jboss | wc -l
Next, check which process is holding most file handles. For example:
$ ps -ef | grep 2873 frances+ 2873 2333 14 10:15 tty2 00:16:14 /usr/lib64/firefox/firefox
Besides it, you should also check what is the status of these file handles. You can verify it with the netstat command. For example, to verify the status of the sockets opened for the process 7790, you can execute the following command:
$ netstat -tulpn | grep 7790 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:54200 0.0.0.0:* LISTEN 7790/java tcp 0 0 127.0.0.1:8443 0.0.0.0:* LISTEN 7790/java tcp 0 0 127.0.0.1:9990 0.0.0.0:* LISTEN 7790/java tcp 0 0 127.0.0.1:8009 0.0.0.0:* LISTEN 7790/java tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 7790/java udp 0 0 224.0.1.105:23364 0.0.0.0:* 7790/java udp 0 0 230.0.0.4:45688 0.0.0.0:* 7790/java udp 0 0 127.0.0.1:55200 0.0.0.0:* 7790/java udp 0 0 230.0.0.4:45688 0.0.0.0:* TIME_WAIT 7790/java udp 0 0 230.0.0.4:45689 0.0.0.0:* TIME_WAIT 7790/java
From the netstat output, we can see a small number of TIME_WAIT sockets, which is absolutely normal. You should worry if you detect thousands of active TIME WAIT sockets. If that happens, you should consider some possible actions such as:
- Make sure you close the TCP connection on the client before closing it on the server.
- Consider reducing the timeout of TIME_WAIT sockets. In most Linux machines, you can do it by adding the following contents to the sysctl.conf file (f.e.reduce to 30 seconds):
net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_fin_timeout = 30 net.nf_conntrack_max = 655360 net.netfilter.nf_conntrack_tcp_timeout_established = 1200
- Use more client ports by setting net.ipv4.ip_local_port_range to a wider range.
- Have the application to Listen for more server ports. (Example httpd defaults to port 80, you can add extra ports).
- Add more client IPs by configuring additional IP on the load balancer and use them in a round-robin fashion.
Increasing the OS limits for your machine
At this point, we know that we have an high number of File handles but there are no leaks in the JVM nor in other processes. Therefore, we have to check if our machine is able to use that amount of File Handles.
On a Linux Box you use the sysctl command to check the maximum number of files your current value:
$ sysctl fs.file-max fs.file-max = 8192
This is the maximum number of files that you can open on your machine for your processes. The default value for fs.file-max can vary depending on your OS version the the amount of physical RAM size at boot.
If you want to change the above limit, edit as root the /etc/sysctl.conf file and set a value for the fs.file-max property. for example:
fs.file-max = 9223372036854775807
Next, to reload the sysctl settings from the default location execute:
sysctl -p
Finally, it is worth mentioning that you can also set a value for the fs.file-max property without editing manually the file. For example:
$ sudo/sbin/sysctl -w fs.file-max=<NEWVALUE>
Monitoring the User limits
At this point, we know that your kernel is not enforcing limits on the number of open files. It can be the case that your current user cannot handle too many files open.
The following error message is an hint that the current limits are set too low for the number of files/sockets needed.
Caused by: java.nio.file.FileSystemException: /opt/wildfly/standalone/deployments: Too many open files at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) at java.nio.file.Files.newDirectoryStream(Files.java:589)
To verify the current limits for your user, run the command ulimit:
$ ulimit -n 1024
To change this value to 8192 for the user jboss, who is running the Java application, change as follows the /etc/security/limits.conf file:
jboss soft nofile 8192 jboss hard nofile 9182
Please be aware that changes in the file /etc/security/limits.conf can be overwritten if you have any file matching /etc/security/limits.d/*.conf on your machine. Please review all these files to make sure they don’t override your settings.
Finally, if you don’t want to set a limit only for a specific user, you can enforce the limit for all users with this entry in /etc/security/limits.conf :
* - nofile 8192
The above lines set the hard limit to 8192 for all users. Reboot the machine for changes to take effect.
Finally please note that the user limits will be ignored when start your process from within a cron job as the cron does not uses a login shell. One potential workaround is to edit the crond.service and increase the TasksMax property accordingly. For example:
Type=simple Restart=no ... TasksMax=8192
Conclusion
In conclusion, “Too many open files” is a common issue that can occur in Java applications, and it can have a significant impact on the performance and stability of your system. However, by understanding the root cause of the problem, and implementing effective solutions such as increasing the file descriptor limit, closing unused files, you can effectively resolve this issue and ensure that your Java applications run smoothly.