Troubleshooting java.net.SocketException: Connection reset

The java.net.SocketException Connection reset is a common Runtime error within Java applications. This tutorial delves into its occurrences, diagnostic steps, and effective solutions.

About “Connection reset” and Possible Causes

A java.net.SocketException Connection reset surfaces during abrupt termination of a network connection. The error often emerges when a party involved in the connection closes it suddenly, deviating from the typical TCP termination process. A sample stack trace could resemble:

java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:235)

For example, if an application closes the socket (ends the TCP session) while there is outstanding data in its receive buffer, the OS sends a connection reset (RST) to the remote side to notify the remote side that the application did not receive all the data.

How to solve the error java.net.SocketException: Connection reset

On the other hand, the error variant, “java.net.SocketException: Connection reset by peer,” indicates the remote host’s transmission of the connection reset.

Here are some common scenarios in which a connection reset (RST) can happen instead of a normal TCP termination:

Common Causes of Connection reset

Client-Initiated RST

  • Error Detection: The client can send an RST packet if it encounters an error during the communication process, such as a network error.
  • Ungraceful Closure: If the client application decides to terminate the connection abruptly without sending the proper FIN (finish) packet, it will send an RST packet to inform the server of the immediate connection termination.

Server-Initiated RST

  • Error Detection: The server can send an RST packet if it detects an error during the communication, such as an invalid SYN packet, or an unexpected packet sequence.
  • Forged Packets: If the server receives a forged SYN or SYN-ACK packet, either from a network scanner or a malicious actor, it will send an RST packet to reject the connection.
  • Client Closure Without FIN: If the client closes the connection without sending the proper FIN packet, the server will detect the incomplete connection and send an RST packet.

Firewall Interference

  • Traffic Blocking: Firewalls can block incoming/outgoing traffic based on various rules, such as IP addresses, port numbers, or protocol types. If a firewall blocks a packet during the TCP handshake or data transfer, it can trigger an RST packet.

Network Scanner Disruptions

  • Forged RST Packets: Network scanners may use forged RST packets to probe or disrupt active TCP connections. These packets can cause the peer to believe the connection has terminated, leading to premature closure and RST exchange.

Network Switch Idle Timeouts

  • TCP Idle Timeout: Network switches with the “TCP idle timeout” feature enabled may prematurely terminate idle TCP connections to conserve resources.

Database Server Issues

  • Database Configuration: Database server configurations, such as timeouts or resource limits, can influence connection stability. In this scenario, the DB may send an RST packet to terminate the connection. See this article: How to validate Database connections in JBoss / WildFly

Finding the Root Cause

Now that we know all potential causes, let’s see how to understand which is the peer which is sending a Connection Reset.

Firstly, check the full logs of every component. For example, here is a log that you could find on WildFly or Tomcat:

ClientAbortException: java.net.SocketException: Connection reset

In this case, The ClientAbortException (or a broken pipe error) means that the client (typically a web browser or HTTP client application) closed the connection prematurely before the server completed the flush of information to the output stream). A common reason for that is that the Client pressed refresh, stop, or closed the Web page.

If the Root cause is not evident from the Stack Trace then you should use some low-level tools such as tcpdump to investigate. Here is a list of commands that can help:

# Detect all packets on localhost and port 5550    
sudo tcpdump -i any host localhost and port 5500

# Detect all tcp packects to port 5500 sending a Connection Reset
sudo tcpdump -i lo 'tcp port 5500 and (tcp[tcpflags] & tcp-rst != 0)'

The first command will produce a more verbose output. However, you will see the sequence of events that causes the Connection reset.

A Session to reproduce a java.net.SocketException Connection reset

To understand better how to find the Root Cause, we will write a simple reproducer for it. We will then look at the tcpdump to show evidence of which layer is causing the issue.

Firstly, let’s code the Java TCP Server:

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;

public class TCPServer {
    public static void main(String[] args) {
        final int PORT = 5500;

        try (ServerSocket serverSocket = new ServerSocket(PORT)) {
            System.out.println("Server listening on port " + PORT);

            while (true) {
                Socket clientSocket = serverSocket.accept();
                System.out.println("New client from " + clientSocket.getInetAddress() + ":" + clientSocket.getPort());
                Thread.sleep(1000); // Sleep for 1 second
                clientSocket.setSoLinger(true, 0);
                clientSocket.close();
            }

        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
        }
    }
}

The setSoLinger(true, 0) call configures the socket to enable the SO_LINGER option with a timeout of zero seconds. Setting the SO_LINGER option to zero means that when the socket is closed, it sends a TCP RST (Reset) packet immediately to forcefully terminate the connection. This can result in a TCP RST being sent to the client, abruptly terminating the connection on the server side without completing the normal TCP connection termination handshake.

Here is the Java TCP Client that connects to it:

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;

public class SimpleClientApp {

    public static void main(String[] args) throws IOException {
        try (Socket socket = new Socket()) {
            socket.connect(new InetSocketAddress("localhost", 5500));
            byte[] buffer = new byte[1024];
            int bytesRead = socket.getInputStream().read(buffer);
            System.out.println(new String(buffer, 0, bytesRead));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Next, activate tcpdump as follows to capture packets on localhost and port 5500:

sudo tcpdump -i any host localhost and port 5500

Finally, run the SimpleClientApp. You will see the java.net.SocketException: Connection reset error:

java SimpleClientApp 
java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
	at java.base/java.io.InputStream.read(InputStream.java:218)
	at SimpleClientApp.main(SimpleClientApp.java:11)

From the tcpdump, you will see evidence that the Server is causing the issue:

java.net.SocketException: Connection reset

Here’s a breakdown of the relevant packets:

  1. 15:45:24.992871: The client sends a SYN (synchronize) packet to the server, initiating the TCP connection.
  2. 15:45:24.992884: The server responds with a SYN-ACK (synchronize acknowledgment) packet, acknowledging the client’s SYN and sending its own SYN, acknowledging the client’s initial sequence number.
  3. 15:45:24.992893: The client sends an ACK (acknowledgment) packet, acknowledging the server’s SYN-ACK. This completes the TCP three-way handshake and establishes the connection.
  4. 15:45:25.993295: The server sends an RST packet to the client, abruptly terminating the connection. The RST flag indicates that the server detected an error or that the client has closed the connection without sending the proper FIN flag.

How do we know that the server is the one sending an RST packet ?

From the last line we can see that localhost.fcp-addr-srvr1 sends to localhost.51048 a RST (R.) packet:

localhost.fcp-addr-srvr1 > localhost.51048  Flags [R.]

In real world scenarios you should see, instead of localhost the IP Address of the server. That would make obvious where the RST packet comes from.

Besides, there is another hint: the destination port (51048) appears random in this case because it’s likely a dynamically assigned port used by the client process for establishing the TCP connection. The server port (5500) is typically the well-known port that the server listens on for incoming connections.

Conclusion

Understanding the causes and diagnostic methods for java.net.SocketException: Connection reset aids in resolving this common Java runtime issue. By employing diagnostic tools and understanding network behaviors, one can effectively identify and rectify these errors.