JBoss AS 7 Performance tuning

In this tutorial you will learn how to tune the application server JBoss AS 7 or the Red Hat’s supported version JBoss EAP 6.

If you need details about the most recent version of the application server, named WildFly, we suggest having a look at this tutorial: WildFly performance tuning

Tuning JBoss application server

Although many architects and software engineer agree that about 70-80% of the application performance depends on how the application itself is coded—a poorly configured server environment can affect your user experience significantly, and eventually, on your application value. The amount of configuration elements, which can influence your server performance are quite a lot; however, some of them deserve a special attention:•    JVM tuning•    Application server resource pooling
•    Logging
•    Caching data

Let’s see each element in detail

JVM tuning

The JBoss AS 7 runs within a Java Virtual Machine (JVM), hence it’s natural that the AS can give a better performance with proper configuration of the JVM parameters.
JVM tuning has evolved over the years and actually changed with each version of Java. Since the  release 5.0 of the J2SE, the JVM is able to provide some default configuration (“Ergonomics”) which is consistent with your environment. However, the smarter choice provided by Ergonomics is not always the optimal and without an explicit user setting, the performance can fall below your expectations.
Basically, the JVM tuning process can be divided into the following steps:
•    Choose a correct JVM heap size. This can be divided into setting an appropriate initial heap size (-Xms) and a maximum heap size (-Xmx).
•    Choose a correct Garbage collector algorithm.

Let’s see both elements more in details.

Choosing the correct JVM heap size

Java objects are created in Heap; each heap is divided into three parts or generations for sake of garbage collection in Java. These are called as Young generation, Tenured or Old generation, and Permanent area of heap.
New Generation is further divided into three parts known as Eden space, Survivor 1, and Survivor 2 space. When an object is first created in heap, it gets created in new generation inside Eden space, and after a subsequent minor Garbage collection if the object survives, it gets moved to survivor 1, and then to Survivor 2 before the major garbage collection moved that object to Old or Tenured generation.
Permanent generation of Heap or Perm Area of Heap is somewhat special and it is used to store meta data related to the classes and methods in JVM; it also hosts String pool provided by JVM.
jboss 7 performance tuning tutorial

In order to tune the JVM, you should choose a correct ratio between young generation (where objects are initially placed after instantiation) and the tenured generation (where old living generations are moved).

Note: For most Java Enterprise applications, the correct ratio between the young generation and the tenured generation ranges between 1/3 and close to 1/2.
The appropriate max heap size can be determined by testing your application with a peak load for a consistent time. Once you have determined the peak of memory demanded by your application, you can allow an extra 25-40% additional maximum heap size, depending on the nature of your application.
As far as it concerns, the initial heap size, a good rule of thumb is to set it  to be the same as the maximum heap size. This increases predictability and avoids  the need to allocate memory to expand the heap. This is particularly useful for production environment, while developers (who have limited resources) might choose a smaller initial heap size.
Keep this suggested configuration as a reference for smaller environments and also for larger ones:

$ java -Xmx1024m -Xms1024m -XX:MaxNewSize=448m -XX:NewSize=448m -XX:SurvivorRatio=6
$ java –Xmx2048m –Xms2048m -XX:MaxNewSize=896m -XX:NewSize=896m -XX:SurvivorRatio=6

Tuning the garbage collector

Garbage collection is a mechanism provided by Java Virtual Machine to reclaim heap space from objects, which are eligible for Garbage collection.
An object becomes eligible for Garbage collection or GC if it is not reachable from any live threads or any static references. In other words, you can say that an object becomes eligible for Garbage collection if all its references are null.
Choosing the correct Garbage collector algorithm is a key (but often overlooked) factor which plays an important role in reaching your service level requirements. There are several garbage collectors available as:
•    Serial collector (-XX:+UseSerialGC): It performs garbage collector using a single thread which stops other JVM threads. This collector is fit for smaller applications; we don’t advice using it for Enterprise applications.
•    Parallel collector (-XX:+UseParallelGC): It performs minor collections in parallel and because J2SE 5.0 can also perform major collections in parallel (-XX:+UseParallelOldGC). This collector is fit for multiprocessor machines and applications requiring high throughput. It is also a suggested choice for applications which produces a fragmented Java heap, allocating large-size objects at different timelines.
•    Concurrent collector (-XX:+UseConcMarkSweepGC): It performs most of its work concurrently using a single garbage collector thread that runs with the application threads simultaneously. It is fit for fast processor machines and applications with a strict service-level agreement. It can be the best choice, also for applications using a large set of long-lived objects live HttpSessions.

Which Garbage Collector is using JBoss AS by default ?

This basically depends on the JDK version you are using, however you can easily find the defaults of your JVM by executing:

$ java -XX:+PrintCommandLineFlags -version

-XX:InitialHeapSize=191460096 -XX:MaxHeapSize=3063361536 -XX:+PrintCommandLineFlags -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC 
java version "1.8.0_60"

The new G1 collector

One of the major enhancements in Java 7 is the new G1 (“Garbage first”) low-latency garbage collector planned to replace CMS in the Hotspot JVM. It is a server-style collector, targeted at multiprocessor machine with large amounts of memory.
The G1 collector is a departure from earlier collectors which had a physical separation between the young and old generations. With G1, even though it is generational, there is no physical separation between the two generations. This collector that divides the entire space into regions and allows a set of regions to be collected, rather than split the space into an arbitrary young and old generation.
The key features of the G1 collector are:

1.    G1 uses parallelism which are mostly used in hardware today. The main advantage of G1 is designed in such a way to make use of all the available CPUs and utilize the processing power of all CPUs as well as increase the performance and speed up the garbage collection.
2.    Next feature which plays a key role in increasing the garbage collection is treating the young objects(newly created) and the old objects (which lived for some time) differently. G1 mainly focuses on young objects as they can be reclaimable when traversing the old objects.
3.    Heap compaction is done to eliminate fragmentation problems. In essence, because G1 compacts as it proceeds, it copies objects from one area of the heap to the other. Therefore, because of compaction, it will not encounter fragmentation issues that CMS might. There will always be areas of contiguous free space from which to allocate, allowing G1 to have consistent pauses over time.

Compared to CMS, the G1 collector is also much easier to use, because it has a lesser number of switches and hence tuning VM is simpler. G1 is already present in JDK 7 as of now and one can try it. To use G1, these two switches need to be passed: -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

Using large memory pages

Another area of the JVM tuning which can yield substantial benefits to your application is the usage of large memory pages. Large memory pages is a feature of modern CPUs that allow memory-hungry applications to allocate memory in 2-4 MB chunks, instead of the standard 4KB. Beginning with Java SE 5.0, there is a cross-platform flag for requesting large memory pages: -XX:+UseLargePages (On by default for Solaris, Off by default for Windows and Linux).

The goal of large-page support is to optimize processor Translation Lookaside Buffer (TLB). A Translation Lookaside Buffer is a page translation cache that holds the most-recently used virtual-to-physical address translation. TLB is a scarce system resource. A TLB miss can be costly as the processor must then read from the hierarchical page table, which may require multiple memory accesses. By using bigger page size, a single TLB entry can represent large memory ranges. There will be less pressure on TLB, and the memory-intensive applications may have better performance.

Large memory pages are available with the 64-bit JVM. (Red Hat Enterprise Linux does let you allocate large pages on the 32-bit OS, but you get an illegal argument when starting the JVM).
The Sun JVM, as well as OpenJDK, requires the following option, passed on the command line, to use large pages: -XX:+UseLargePages.
Application server resource pools
Application server pools are used because the very first release of any application server as means to set boundaries for the resources they contain.
Resource pooling offer several benefits, like:
•    Improved performance: You can re-assign resource-intensive objects such as a database connection instead of creating and destroying them  every time.
•    Improved security: By granting a limited number of resources, you prevent plundering of server resources from applications which could eventually lead to an interruption of the AS services.

JBoss AS 7 uses several resource pools to manage different kind of services. The application server ships with a default configuration for all resource pools which could be just good for simple applications. If you are planning to write mission-critical applications, however, you need to find the appropriate number of resources to be assigned to your pools.
We will discuss in particular on the following pool of resources, which ultimately play an important role on performance tuning:
•    The database connection pool
•    The EJB pool used by Stateless EJBs and MDBs
•    The Web server pool of threads

This version of the application server is not able to produce performance metrics for all the  subsystem which we have mentioned. Although, it would be preferable to monitor the application server pools through management interfaces, you can still have a look inside the application server pools using some other tools or with a minimal sample application. That’s what we will do in the next sections. (We’ll however update this article as soon as the performance tuning metrics are available so come back to visit us soon!)

Tuning the database connection pool

Establishing a JDBC connection with a DBMS can be quite slow. If your application requires database connections that are repeatedly opened and closed; this can become a significant performance issue. The connection pools in JBoss AS datasources offer an efficient solution to this problem.
What is important to stress out is that, when a client closes a connection from a data source, the connection is returned to the pool and becomes available for other clients; therefore, the connection itself is not closed. The cost of opening and closing pooled connections can be measured in terms of nanoseconds, so it’s irrelevant in terms of performance.
In the following example, we are enhancing the datasource configuration exposed in Chapter 3, Configuring Enterprise Services with some connection pool configuration:

<datasource jndi-name="MySqlDS" pool-name="MySqlDS_Pool"
    enabled="true" jta="true" use-java-context="true" use-ccm="true">

Here, we configured an initial pool capacity of ten connections which can grow up to thirty. As you can see from the following MySQL administration console, when you set the pre-fill element to true, the application server attempts to pre-fill the connection pool at the start-up. This can produce a performance hit, especially if your connections are costly to acquire.

jboss 7 performance tuning tutorial
If the application server is not able to serve any more connections because they are all in use, then it will wait up to the blocking-timeout-millis before throwing an exception to the client.
At the same time, connections which have been idle for some minutes over the parameter idle-timeout-minutes, they are forced to return to the pool.

Adjusting the pool size

To determine the proper sizing, you need to monitor your connection usage. This can be done in several ways. If you have access to the Command Line Interface you can monitor the runtime properties of your datasource. Here’s a sample output of our example application delivered in chapter 4:

"outcome" => "success",
"result" => {
"ActiveCount" => "10",
"AvailableCount" => "29",
"AverageBlockingTime" => "0",
"AverageCreationTime" => "56",
"CreatedCount" => "10",
"DestroyedCount" => "0",
"MaxCreationTime" => "320",
"MaxUsedCount" => "5",
"MaxWaitCount" => "0",
"MaxWaitTime" => "1",
. . . .

The output of this command is quite verbose however the most interesting attributes are located at the beginning of the output: in particular the ActiveCount attribute which displays the amount of connections which are currently active and the MaxUsedCount which is the peak of connections used by the application.
Beware: if you are prefilling the pool, as shown in the earlier section, these connection will all result Active. This could be misleading and lead you to assume they are actually busy. 
If you are not able to use the CLI or simply you want to make good use of your DBA certification there are some valid alternative as well: the first and most obvious is monitoring the database sessions. The following table shows some useful commands, which can be used to keep track of active database connections on different databases:

Database Command / Table
Oracle Query the V$SESSION view
Postgre-SQL Query the PG_STAT_ACTIVITY table

Another option is using a tool like P6Spy which acts as a JDBC proxy driver. (I’ve blogged an article about it here).
Once you have found the peak of connection used by your application, just set the maximum at least 25-30% higher. Don’t be concerned about setting the maximum too high, because if you don’t need that many connections, the pool will shrink automatically, provided that you have set idle-timeout-minutes.
On the other hand, your server logs are still an invaluable help to check if your pool is running in trouble. For example, if you start seeing this exception in your server logs, there is a strong clue that you need to look at your connection pooling:

21:57:57,781 ERROR [stderr] (http-executor-threads – 7) Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
21:57:57,782 ERROR [stderr] (http-executor-threads – 7)         at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection

 Tuning JBoss AS 7 EJB connection pool

The creation and destruction of beans can be an expensive operation, especially if they acquire external resources. To reduce this cost, The EJB container creates pool of beans which, therefore, don’t need to be re-initialized every time they are needed.
The Stateless EJB pool and MDB pool are used to provide stateless business services to their client, acquiring beans from the pool when they are requested and releasing the bean to the pool as soon as they are finished.
A typical EJB pool configuration looks like the following:

   <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
   <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

At the time of writing in AS 7, there is only support for strict-max-pool as a bean instance pool.
A strict max pool allows you to configure a maximum upper limit for the pool. At runtime, when all the bean instances from the pool are in use and a new bean invocation request comes in, the pool blocks the request till the next bean instance is available or till a timeout (set in instance-acquisition-timeout) is reached.

 Monitoring the EJB pools will be soon available through the CLI, which will have a set of minimal operations to check the pool metrics. Setting up a self-made solution to monitor the current pool size is, however, not too complicated: you can use the handy EJB3 Interceptor API to monitor the EJB which have been taken from the pool and those which have been released.

In the following Interceptor, we are simply setting a single EJB Singleton field before contacting the EJB, and after that, it has completed its job and hence returned to the pool.

package com.packtpub.chapter12;
import javax.ejb.EJB;
import javax.interceptor.AroundInvoke;
import javax.interceptor.InvocationContext;

public class EJBInterceptor {
    EJBCounter singleton;

    public Object defaultMethod(InvocationContext context) throws
        // Take a bean from the pool

        // Invoke the EJB method
        Object result = context.proceed();

        // Return the bean to the pool

        // Prints out the current pool size
        return result;

The EJBCounter is a singleton EJB which merely contains the counter variable, holding the EJB max-pool-size.

package com.packtpub.chapter12;

import javax.ejb.Singleton;

public class EJBCounter {
    private int count=20;

    public void getFromPool() {
    public void returnToPool() {
    public void dumpPoolSize() {
        System.out.println("Current pool size is "+count);

You can further refine this approach by adding a @PostContruct annotation which loads this variable from an external source such as the DB or a property file.
Another viable option is launching a CLI script which collects the value from the max-pool-size attribute. Here’s an example:

"outcome" => "success",
"result" => 20

The interceptors can be then applied to the stateless EJB which are used by your application either by means of a simple annotation or by declaring them into the ejb-jar.xml configuration file. For example, here’s how to intercept all EJB invocations via the @Interceptors annotation:

import javax.ejb.Stateless;
import javax.interceptor.Interceptors;

public class EJBTest {
. . .

Turning off the EJB pool

Although this might sound weird to you, there can be some scenarios where you don’t want your EJB resources to be managed by a pool but created on demand. For example, if your EJB does not need a costly initialization (like acquiring an external resources), it can be advantageous, in terms of performance, to avoid using the EJB 3 pool. (the JBoss 5 Performance tuning book, for example, shows a case where a so-called heavy Stateful EJB can even outperform the Stateless counterpart. This is mostly due to the fact that handling the stateless pool is not a trivial task in terms of performance).
Switching off the EJB pool just requires to comment/evict the bean-instance-pool-ref element which refers to the EJB pool:

 <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

Of course, it is strongly recommended to run a consistent test bed to demonstrate that your application can benefit from such a change, which will take away any check on the number of EJB instances.

 Tuning Web server thread pool

There are a large set of tuning aspects which ultimately influence the performance of the web server. One of the most important factors is tuning the HTTP Connector thread pool settings to more closely match the web request load you have. This is difficult to do, but is very important to get right for best performance.
The amount of threads which are allowed by the web server are referenced through the executor attribute:

<subsystem xmlns="urn:jboss:domain:web:1.0">

 <connector enable-lookups="false" enabled="true"        
 max-post-size="2048" max-save-post-size="4096" 
 name="http" protocol="HTTP/1.1" 
 proxy-name="proxy" proxy-port="8081" 
 redirect-port="8443" scheme="http" 
 secure="false" socket-binding="http" />
. . .

Then, within the threads subsystem, you can define the number of threads which will be used by the pool, along with the other thread attributes (see Chapter 2, Configuring the Application Server, for more details about the thread subsystem):

<subsystem xmlns="urn:jboss:domain:threads:1.0">
    <bounded-queue-thread-pool name="http-executor"
        <core-threads count="10" per-cpu="20" />
        <queue-length count="10" per-cpu="20" />
        <max-threads count="10" per-cpu="20" />
        <keepalive-time time="10" unit="seconds" />

The most important Connector attributes are defined into the core-threads and max-threads. Setting these values too low means that you may not have enough threads to handle all of the requests, in which case, requests have to sit idle for some time without being handled until another request thread is freed up. Too low of a value also means that JBoss Web server will be unable to take advantage of your server machine’s hardware.
On the other hand, be careful before increasing these thread counts blindly. By increasing the thread count too much, you will:
•    Consume a good chunk of memory
•    Your system will spend too much time-context switching

You should, at first, investigate if it’s rather a problem of individual requests taking too long. Are your threads returning to the pool? If, for example, database connections are not released, threads pile up waiting to obtain a database connection thereby making it impossible to process additional requests.
In such a scenario, simply adding more thread will make things even worse by introducing a greater stress on the CPU and on the garbage collector. You can discover this kind of problem by simply taking a thread dump in your application to find out where your web server threads are stuck. For example, in this picture, taken from JConsole threads tab, you can see how an idle thread should look like, by looking at its stack trace:
jboss 7 performance tuning howto book

On the other hand, the following HTTP thread is busy at doing input/output operations which could mean, for example, the web server is acquiring data from an external resource.

jboss 7 book performance tuning
The above snapshots gives you also a clue on how you can monitor the number of web server running threads. Just fill in the executor name (http-executor) in the lower textfield, and you will have a filtered list of all your web server threads.

JBoss AS 7 Logging tuning

Logging is an essential activity of every applications, however the default configuration is generally appropriate for development, but not for a production environment.
The key elements which you need to consider when switching to production are:

1.    Choosing the appropriate handler to output your logs.
2.    Choose a log level which provides just the amount of information you need and nothing else.
3.    Choose an appropriate format for your logs

As far as it concerns, log handlers, in the default configuration, both console logging and file logging are enabled. While this can be fine for development, using console logging in production is an expensive process which causes lots of un-buffered I/O. While some applications maybe fine with console logging, high-volume applications benefit from turning off console logging and just using the FILE handler.
In order to remove console logging, you can simply comment out its handler:

  <level name="INFO"/>
     <!--   <handler name="CONSOLE"/> -->
   <handler name="FILE"/>

Next step is choosing the correct logging verbosity. Obviously, the less you log, the less I/O will occur, and the better your overall application. The default configuration uses the “INFO” level for the root logger. You could consider raising this to an higher threshold like “WARN” or (using a fine grained approach) changing the single logging categories

<logger category="org.hibernate">
  <level name="WARN"/>

In this example, we have just raised the log level for org.hibernate package to “WARN” which will produce a much more concise information from Hibernate.
Finally, also the pattern used by your logs can influence the performance of your applications. For example, let’s take the default pattern format, which is:

 pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

A detailed explanation of the logging patterns can be found in Chapter 2’s section The console handler. Starting from this basic format, with as little as adding the flag %l, you can greatly enhance the verbosity of your logs by printing the line number and the class that emitted the log:

 pattern="%l %d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

And this is the console output once the server configuration has been reloaded:

jboss 7 book performance tuning
While this information can be quite useful in development, it will result in a huge burden when ported in production.
The other flags which can have a negative impact on your logging performance are %C (which prints out the caller class information) , %M (which outputs the method where logging was emitted) and %F (which outputs the filename where the logging request was issued).

Cache tuning

Most performance issues in Enterprise applications arise from data access, hence caching data is one of the most important tuning techniques.

The current release of the application server uses Infinispan as distributed caching provided and you can use it to cache anything you like.
In particular, you can use it as second-level caching provider by adding the following configuration in your persistence.xml file:


 <property name="hibernate.cache.use_second_level_cache" value="true"/>
 <property name="hibernate.cache.use_minimal_puts" value="true"/>

Second-level caching is intended for data that is read-mostly. It allows you to store the entity and query data in memory so that this data can be retrieved without the overhead of returning to the database.
On the other hand, for applications with heavy use of write operations, caching may simply add overhead without providing any real benefit.
In order to cache entities, you can use the @javax.persistence.Cacheable in conjunction with the shared-cache-mode element of persistence.xml. When you have enabled a selective cache of your entities, the @Cachable annotation will load entities into the Hibernate second-level cache.
If you want to monitor the cache statistics, you can use the following property in your persistence.xml file, which will expose the cache statistics via JMX:

<property name="hibernate.cache.infinispan.statistics" value="true"/>

A very simple way to check the MBeans exposed by AS 7 is starting the JConsole application and choosing the Mbeans tab in the upper area of the application:

jboss 7 performance tuning book
The following table describes synthetically the meaning of the statistics provided by Infinispan’s cache:

Attribute Description
Evictions Number of cache eviction operations
RemoveMisses Number of cache removals where keys were not found
ReadWriteRatio Read/writes ratio for the cache
Hits Number of cache attribute hits
NumberofEntries Number of entries currently in the cache
StatisticsEnabled Enables or disables the gathering of statistics by this component
TimeSinceReset Number of seconds since the cache statistics were last reset
ElapsedTime Number of seconds since cache started
Misses Number of cache attribute misses
RemoveHits Number of cache removal hits
AverageWriteTime Average number of milliseconds for a write operation in the cache
Stores Number of cache attribute put operations
HitRatio Percentage hit/(hit+miss) ratio for the cache
AverageReadTime Average number of milliseconds for a read operation on the cache

Evicting data from the cache is also fundamental in order to save memory when cache entries are not needed anymore. You can configure the cache expiration policy, which determines when the data will be refreshed in the cache (for example, 1 hour, 2 hours, 1 day, and so on) according to the requirements for that entity.

Configuring data eviction can be done either programmatically (see Infinispan documentation for examples about it: https://docs.jboss.org/author/display/ISPN/Home) or in your configuration file. For example, here’s a sample configuration which could be added to your persistence.xml to configure data eviction:

<property name="hibernate.cache.infinispan.entity.eviction.strategy"
 value= "LRU"/>
<property  name="hibernate.cache.infinispan.entity.eviction.wake_up_interval" value= "2000"/>
<property name="hibernate.cache.infinispan.entity.eviction.max_entries"
 value= "5000"/>
<property name="hibernate.cache.infinispan.entity.expiration.lifespan"
 value= "60000"/>
<property name="hibernate.cache.infinispan.entity.expiration.max_idle"
 value= "30000"/>

And here’s a description for the properties which are contained in the configuration file:

Property Description
hibernate.cache.infinispan.entity.eviction.strategy The eviction strategy used by Infinispan. Can be either UNORDERED, FIFO, LIFO, NONE.
hibernate.cache.infinispan.entity.eviction.wake_up_interval The time (ms) interval between each eviction thread runs.
hibernate.cache.infinispan.entity.eviction.max_entries The maximum number of entries allowed in a cache (after that, eviction takes place).
hibernate.cache.infinispan.entity.expiration.lifespan The time expiration (ms) of entities cached.
hibernate.cache.infinispan.entity.expiration.lifespan The time expiration (ms) of entities cached.

Hungry for Tuning?

     Speed-up your Enterprise Applications with our WildFly Performance Tuning guide!

Other tuning resources:

JBoss Performance Tuning part 1

JBoss performance tuning part 2

WildFly performance tuning

Web services performance tuning