Detecting Connection leaks with WildFly

In this article, we will explore how to detect Database Connection leaks in WildFly using several management instruments. At the end of it, you will be able to identify and resolve your connection leaks effectively.

Understanding Connection Leaks

Before delving into the tools to detect connection leaks, it’s important to understand what connection leaks are. In the context of database connections, a leak occurs when an application fails to release a connection back to the pool after it has finished using it. Over time, these leaked connections accumulate, leading to a depletion of available connections and eventual system instability.

The following article will teach you the safe way to close Database connections: Using try-with-resources to close database connections

Using IronJacamar LeakDumperManagedConnectionPool

IronJacamar is a Java Connector Architecture (JCA) implementation included in WildFly, responsible for managing connection pools and resource adapters. LeakDumperManagedConnectionPool is a specialized component within IronJacamar that aids in detecting connection leaks.

In order to detect Connection leaks you have to add a couple of properties to your startup script. When using Windows:

set "JAVA_OPTS=%JAVA_OPTS% -Dironjacamar.mcp=org.jboss.jca.core.connectionmanager.pool.mcp.LeakDumperManagedConnectionPool -Dironjacamar.leaklog=leaks.txt"

When using Linux:

JAVA_OPTS="$JAVA_OPTS -Dironjacamar.mcp=org.jboss.jca.core.connectionmanager.pool.mcp.LeakDumperManagedConnectionPool -Dironjacamar.leaklog=leaks.txt"

The above properties will acttivate the LeakDumperManagerConnectionPool which will trace the leaks in the file named leaks.txt.

In order to trigger the LeakDumper, you need to flush the Connections in the pool, once that you have a suspect of a Connection Leak:

/subsystem=datasources/data-source=PostgreDS/:flush-all-connection-in-pool

Now check the file leaks.txt which contains the Stacktrace from Classes that are producing a Connection Leak:

Leak detected in pool: PostgreSQL
  ConnectionListener: 1c6c192
  Allocation timestamp: 1438520410260
  Allocation stacktrack:
java.lang.Throwable: ALLOCATION LEAK
        at org.jboss.jca.core.connectionmanager.pool.mcp.LeakDumperManagedConnectionPool.getConnection(LeakDumperManagedConnectionPool.java:96)
. . . . . org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:67)
		at com.sample.LeakServlet.processRequest(LeakServlet.java:40)
		at com.sample.LeakServlet.doGet(LeakServlet.java:72)

As you can see we have got an interesting information: the leak was produced by the com.sampleLeakServlet in the method processRequest (line 40) and it was reported at the timestamp indicated above.

By the way, you can convert the timestamp in an human readable date with few rows of Java code:

Timestamp stamp = new Timestamp(1438520410260);
Date date = new Date(stamp.getTime());
System.out.println(date);

Using Cached Connection Manager to identify a Leak

Another option to identify a Connection Leak consists in setting the debug attribute of the Cached Connection Manager to true. For example:

/subsystem=jca/cached-connection-manager=cached-connection-manager:write-attribute(name=debug,value=true)

When you set the debug attribute to true, the following actions will happen:

  1. WildFly will log an INFO message stating “Closing a connection for you. Please close them yourself.” This message serves as a reminder that you should fix the connection leak in your code.
  2. Generate and log a stacktrace that pinpoints the location in the code where you have the connection leak. This information helps developers identify the exact section of code responsible for the connection leak.
  3. Close the leaked connection. Finally, WildFly will take the necessary action to close the leaked connection, ensuring that the resource is released and available for future use.

By performing these actions, WildFly cached connection managers actively address connection leaks, promoting efficient resource management and preventing performance degradation.

Detecting Connection Leaks from an Heap Dump

If you have an Heap Dump of your application, then you can use OQL (Object Query Language) to analyze and query the Heap Dump data. OQL provides a powerful and flexible way to extract meaningful information from heap dumps by querying the object graph. It allows developers and analysts to perform various operations, such as filtering, grouping, and aggregating objects based on specific criteria.

By executing the following query, you can check the list of Objects in the Pool of the application server:

select p.pool.poolName.toString() AS poolName,
    p.checkedOutSize.value AS inUse,
    p.cls.baseCount AS active,
    p.poolConfiguration.maxSize.value.toString() AS maxSize,
    p.poolConfiguration.minSize.value.toString() AS minSize,
    p.poolConfiguration.strictMin.value.toString() AS useStrictMin,
    p.poolConfiguration.prefill.value.toString() AS prefill
from org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool p

Conclusion
Detecting and addressing connection leaks is crucial for maintaining the performance, stability, and resource efficiency of applications deployed on WildFly. With the help of the IronJacamar LeakDumperManagedConnectionPool and Cached Connection Manager, developers have a powerful tool at their disposal to identify and resolve connection leaks effectively.