JBoss Clustering a Web Application

This article discusses the required steps to cluster a Web application using WildFly application server and the older server releases (JBoss AS 5).

Clustering allows us to run an applications on several parallel servers (a.k.a cluster nodes). The load is distributed across different servers, and even if any of the servers fails, the application is still accessible via other cluster nodes. Clustering is crucial for scalable enterprise applications, as you can improve performance by simply adding more nodes to the cluster.

Clustering Web applications with WildFly

WildFly clustering service is an on-demand service. This means that, even if you have started a cluster aware configuration, the cluster service won’t start until you deploy a cluster-aware application. Enabling clustering for your applications can be achieved in different ways depending on the type of application you are going to deploy:

To cluster a Web based application, all you need is to follow these steps:

  1. Start WildFly with an “ha” profile (or “full-ha”)
  2. Declare your application as “distributable” in the web.xml configuration file to have your HTTP session state replicated across the cluster:
<web-app>
  <distributable />
</web-app>

By default, Web applications rely on a distributed Infinispan Cache:

<cache-container name="web" default-cache="dist" marshaller="PROTOSTREAM" modules="org.wildfly.clustering.web.infinispan">
    <transport lock-timeout="60000"/>
    <replicated-cache name="sso">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="BATCH"/>
    </replicated-cache>
    <replicated-cache name="routing"/>
    <distributed-cache name="dist">
        <locking isolation="REPEATABLE_READ"/>
        <transaction mode="BATCH"/>
        <file-store/>
    </distributed-cache>
</cache-container>

You can further specialize the way HTTP Session is managed through the distributable-web subsystem, which has been introduced in WildFly 17 to manage a set of session management profiles that encapsulate the configuration of a distributable session manager:

<subsystem xmlns="urn:jboss:domain:distributable-web:2.0" default-session-management="default" default-single-sign-on-management="default">

The HTTP Session Management is handled under the hoods by Infinispan therefore if you want to check its settings, you have to reach the infinispan-session-management attribute under the distributable-web subsystem. Example:

 /subsystem=distributable-web/infinispan-session-management=default:read-resource
{
    "outcome" => "success",
    "result" => {
        "cache" => undefined,
        "cache-container" => "web",
        "granularity" => "SESSION",
        "affinity" => {"primary-owner" => undefined}
    }
}

As you can see, there are several configurable attribute for the infinispan-session-management:

  • cache-container: This references the Infinispan cache-container into which session data will be stored.
  • cache: This references a cache for the related cache-container. If undefined, the default cache of the associated cache container will be used.
  • granularity: This defines the session manager mapping for the individual cache entries:
  • affinity: This attribute defines the affinity that a web request should have for a given server.

Configuring HTTP Session Granularity

The granularity attribute can have the following values:

  • SESSION: Stores all session attributes within a single cache entry. This is generally more expensive than ATTRIBUTE granularity, but preserves any cross-attribute object references.
  • ATTRIBUTE: Stores each session attribute within a separate cache entry. This is generally more efficient than SESSION granularity, but does not preserve any cross-attribute object references.

By default, WildFly’s distributed session manager uses SESSION granularity, meaning that all session attributes are stored within a single cache entry. While this ensures that any object references shared between session attributes are preserved following replication/persistence, it means that a change to a single attribute results in the replication/persistence of all attributes.

If your application does not share any object references between attributes, users are strongly advised to use ATTRIBUTE granularity. Using ATTRIBUTE granularity, each session attribute is stored in a separate cache entry. This means that a given request is only required to replicate/persist those attributes that were added/modified/removed/mutated in a given request. For read-heavy applications, this can dramatically reduce the replication/persistence payload per request. Here is how you can set the granularity to “ATTRIBUTE” for the default session manager:

/subsystem=distributable-web/infinispan-session-management=default/:write-attribute(name=granularity,value=ATTRIBUTE)

Configuring HTTP Session Affinity

The affinity attribute defines the affinity that an HTTP request has for a given WildFly server. The affinity of the associated web session determines the algorithm for generating the route to be appended onto the session ID (within the JSESSIONID cookie, or when encoding URLs). Possible values are:

  • affinity=none: HTTP requests won’t have affinity to any particular node.
  • affinity=local: HTTP requests will have an affinity to the server that last handled a request for a given session. This is the standard sticky session behavior.
  • affinity=primary-owner: HTTP requests will have an affinity to the primary owner of a given session. This is the default setting. Behaves the same as affinity=local if the backing cache is not distributed nor replicated.
  • affinity=ranked: HTTP requests will have an affinity for the first available member in a list that include the primary and the backup owners, and for the member that last handled a session. Behaves the same as affinity=local if cache is not distributed nor replicated.

The ranked affinity supports the following attributes:

  • delimiter: The delimiter used to separate the individual routes within the encoded session identifier.
  • max routes: The maximum number of routes to encode into the session identifier.

Here is how to set HTTP Session affinity to be ranked, with max-routes set to 2:

/subsystem=distributable-web/infinispan-session-management=default/affinity=ranked:add(max-routes=2)

Clustering Web applications with JBoss AS 5

The JBoss Application Server (AS) comes with clustering support out of the box.

The simplest way to start a JBoss server cluster is to start several JBoss instances on the same local network, using the run -c all command for each instance. Those server instances, all started in the all configuration, detect each other and automatically form a cluster.

For Web applications, a hardware- or software-based HTTP load-balancer usually sits in front of the application servers within a cluster. These load-balancers can be used to decrypt HTTPS requests quickly, distribute the load between nodes in the cluster, and detect server failures. The usual setup is to have a user session live entirely on one server that is picked by the load-balancer. This is called a “sticky” session.

Configuring JBoss to run in a cluster

So let’s see how to configure 2 instances of JBoss in a cluster. We’ll define two nodes nodeA and nodeB

Node Name Address
nodeA 192.168.0.1
nodeB 192.168.0.2

Now for every installation of JBoss go under the folder deploy\jboss-web.deployer (jBoss 4.X) or under the folder deploy\jbossweb.sar (jBoss 5.X and later) and pick up the file server.xml  line.

Configure on the 192.168.0.1 machine

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="nodeA">

and on the machine 192.168.0.2

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="nodeB">

Now create an HelloWorld web application and add a Servlet which simply dumps some session information:

public void doGet(HttpServletRequest request, HttpServletResponse response)

 throws ServletException, IOException { 
 
  response.setContentType("text/html"); 
  String dummy = (String)request.getSession().getAttribute("dummy"); 
  PrintWriter out = response.getWriter(); 
 
  System.out.print("Hello from Node "+ InetAddress.getLocalHost().toString()); 
 
  if (dummy == null) {  
   out.println("Dummy value not found. Inserting it....");  
   request.getSession().setAttribute("dummy", "dummyOk"); 
  } else {  
   out.println("Dummy is " +dummy); 
  } 
  out.flush(); 
  out.close();
}

Before packing the application add at the very beginning of the web.xml file

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.4"
xmlns="http://java.sun.com/xml/ns/j2ee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">

   <distributable/>
</web-app>

Ok so now you’re ready to deploy the web application: where can I deploy it ?

The easiest way to deploy an application into the cluster is to use the farming service. That is to hot-deploy the application archive file (e.g., the EAR, WAR or SAR file) in the all/farm/ directory of any of the cluster member and the application is automatically duplicated across all nodes in the same cluster.

You can only put archive files, not exploded directories, in the farm directory. This way, the application on a remote node is only deployed when the entire archive file is copied over. Otherwise, the application might be deployed (and failed) when the directory is only partially copied.

If you want to read more about the Farming Service continue reading this article:
http://www.mastertheboss.com/en/jboss-application-server/146-jboss-farming-service.html

Ok, so now Jboss is ready to accept Cluster Web Applications, now we need at first something something sitting in front of our JBoss Cluster. We’ll use for our purpose Apache Web Server.

For a detailed explanation of all steps necessary to configure Apache-JBoss via modJK we suggest you reading the Wiki http://wiki.jboss.org/wiki/UsingMod_jk1.2WithJBoss. Now we’ll focus in particular on the part of the configuration relative to the cluster.

The first thing we need to have is redirecting calls to the mod_JK load balancer: this is done in file mod_jk.properties:

# Where to find workers.properties
JkWorkersFile conf/workers.properties

# Where to put jk logs
JkLogFile logs/mod_jk.log

JkMount /HelloWorld/* loadbalancer

Here HelloWorld is the Web Context of our sample application. Now let’s move to workers.properties file.

worker.list=loadbalancer,status

worker.nodeA.port=8009
worker.nodeA.host=192.168.0.1
worker.nodeA.type=ajp13
worker.nodeA.lbfactor=1

worker.nodeB.port=8009
worker.nodeB.host=192.168.0.2
worker.nodeB.type=ajp13
worker.nodeB.lbfactor=1

worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=nodeA,nodeB

# Status worker for managing load balancer
worker.status.type=status

As you can see we have the two nodes (nodeA and nodeB) with relative port and IP address.

Now launch the two instances of JBoss using the command run -c all. The first instance started will display this information when the second instance kicks in:

16:09:20,562 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-192.168.0.1-8009
16:09:20,578 INFO [Server] JBoss (MX MicroKernel) [4.2.0.GA (build: SVNTag=JBoss_4_2_0_GA date=200705111440)] Started in 1m:20s:438ms
16:10:28,343 INFO [TreeCache] viewAccepted(): [192.168.0.1:2300|1] [192.168.0.1:2
300, 192.168.0.2:2321]
16:10:28,421 INFO [TreeCache] locking the subtree at / to transfer state
16:10:28,593 INFO [StateTransferGenerator_140] returning the state for tree roo
ted in /(1024 bytes)
16:10:40,921 INFO [DefaultPartition] New cluster view for partition DefaultPart
ition (id: 1, delta: 1) : [192.168.0.1:1099, 192.168.0.2:1099]
16:10:40,984 INFO [DefaultPartition] I am (192.168.0.1:1099) received membership
Changed event:
16:10:40,984 INFO [DefaultPartition] Dead members: 0 ([])
16:10:40,984 INFO [DefaultPartition] New Members : 1 ([192.168.0.2:1099])
16:10:40,984 INFO [DefaultPartition] All Members : 2 ([192.168.0.1:1099, 192.168.0.2:1099])
16:10:45,187 INFO [TreeCache] viewAccepted(): [192.168.0.1:2309|1] [192.168.0.1:2 309, 192.168.0.2:2336]
16:10:47,453 INFO [TreeCache] viewAccepted(): [192.168.0.1:2313|1] [192.168.0.1:2 313, 192.168.0.2:2341]

Ok. In order to invoke our Servlet simply point to our Apache Web Server domain address

http://apache.web.server/HelloWorld/helloWorld

You’ll see that the balancer load balances calls and, once it has created the session, it will replicate the session to the secondary server. You can do experiments to test your cluster, switching off one instance of the cluster and verifying that the session stays alive on the other server.

Simply verify that the dummy values stays in the Session when you switch off the current instance of JBoss.

Dummy is dummyOk

Conclusion

JBoss clustering is designed to be simple to configure and simple to extend. It is one of the final pieces of the puzzle that makes JBoss a viable open source competitor to other enterprise J2EE offerings.

You can read more about Cluster Http replication in this tutorial.

How to monitor and invalidate HTTP Sessions in WildFly

This article has been updated to show how to monitor and invalidate HTTP Sessions in WildFly application server / JBoss EAP using management instruments.

First of all, in order to gather statistics about HTTP sessions, you need to enable statistics on undertow subsystem You can do it from the CLI as follows:

/subsystem=undertow:write-attribute(name=statistics-enabled,value=true)

That being said, let’s start checking an example application which has been deployed and has 2 active sessions:

/deployment=example.war/subsystem=undertow:read-resource(include-runtime=true)
{
    "outcome" => "success",
    "result" => {
        "active-sessions" => 2,
        "context-root" => "/example",
        "expired-sessions" => 0,
        "highest-session-count" => 2,
        "max-active-sessions" => -1,
        "rejected-sessions" => 0,
        "server" => "default-server",
        "session-avg-alive-time" => 0,
        "session-max-alive-time" => 0,
        "sessions-created" => 2,
        "virtual-host" => "default-host",
        "servlet" => undefined,
        "websocket" => undefined
    }
}

In order to invalidate HTTP Sessions or check the Session attributes, we need to collect its Session id. This can be done programmaticaly at some point:

HttpSession session = request.getSession();
String sessionid = session.getId();

However you can list HTTP Sessions from the CLI as well:

/deployment=example.war/subsystem=undertow:list-sessions()
{
    "outcome" => "success",
    "result" => [
        "iD_2bXgNFzOa5vPQvDoDuwXr1cec94xn2k-OvRk6",
        "4Jj6Y9qem8vAm8WXT3UAqvApBBfFszMLLWRyUqrK"
    ]
}

With that value, we can invalidate the HTTP Session as follows:

/deployment=example.war/subsystem=undertow:invalidate-session(session-id=iD_2bXgNFzOa5vPQvDoDuwXr1cec94xn2k-OvRk6)

We can also check the list of attributes which have been added to the Session:

 /deployment=example.war/subsystem=undertow:list-session-attribute-names(session-id=k3Yp2uwWYDxWhCe7JhpKghrEsxE6Gyo9lZgFySWC)
{
    "outcome" => "success",
    "result" => ["name"]
}

That can be combined with the attribute’s values:

[standalone@localhost:9990 /] /deployment=example.war/subsystem=undertow:list-session-attributes(session-id=k3Yp2uwWYDxWhCe7JhpKghrEsxE6Gyo9lZgFySWC)
{
    "outcome" => "success",
    "result" => [("name" => "Frank")]
}

Monitoring and invalidating HTTP Sessions from WildFly Web Console

It is worth mentioning, that you can also collect statistics and invalidate HTTP Sessions from the Web Console.

You should select your deployment unit under “Runtime / Server / Web / Deployment”. The following UI shows the core HTTP Session statistics:

To invalidate a HTTP Session, choose a deployment and select the “View” button. Within the available catgories (Sessions/Servlets/WebSockets), choose “Sessions“.

There, you can see all active sessions. Select a session to inspect HTTP session attributes. Also, you can invalidate a session by pressing “Invalidate Session“.

 

That’s it.

Configuring WildFly as Load Balancer for a Cluster

This tutorial is about Load Balancing a cluster of WildFly servers using a WildFly front-end server configured using the load-balancer profile.

Since WildFly 9, you can use an instance of the Application Server as a mod_cluster front-end for your back-end applications. This removes the need to use a native Web server like Apache (and mod_cluster libs installed on it) as load balancer to a cluster of WildFly servers.

Here is a sample view of a cluster of WildFly backend servers fronted by a WildFly frontend server configured to route request using the Mod_cluster Management Protocol (MCMP):

You can configure a WildFly Load Balancer using one of the following options:

  1. Configure the Load Balancer to use Multicast (default)
  2. Configure a static list of Proxies (Load Balancers)
  3. Configure undertow server as static load balancer (reverse proxy)

The common requirements to all configurations are the following ones:

  • You need to configure the cluster nodes with an ha / full-ha profile for your cluster to work.
  • Besides it, make sure that you are deploying a cluster-aware application on the top of WildFly. For example, an application which includes in the web.xml file
<distributable />

Now let’s see these options in detail.

Configuring the Load Balancer using Multicast

This is the simplest option and it will work out of the box if you are using WildFly 10 or newer of JBoss EAP 7. Simply make sure the WildFly Load Balancer uses the “loadbalancer” profile and the cluster nodes the “ha” or “full-ha” profile.

Here is an excerpt from domain.xml to set up Load Balancing in your WildFly Domain:

<server-groups>
    <server-group name="balancer-group" profile="load-balancer">
        <jvm name="default">
            <heap size="64m" max-size="512m"/>
        </jvm>
        <socket-binding-group ref="load-balancer-sockets"/>
    </server-group>
    <server-group name="application-group" profile="ha">
        <jvm name="default">
            <heap size="64m" max-size="512m"/>
        </jvm>
        <socket-binding-group ref="ha-sockets"/>
        <deployments>
            <deployment name="myapp.war" runtime-name="myapp.war"/>
        </deployments>
    </server-group>
</server-groups>

Notice the profile settings for your Domain. Also, an application named “myapp.war” has been deployed on the application-group.

This configuration is paired (host.xml) by the following server definitions:

<servers>
    <server name="balancer" group="balancer-group" auto-start="true"/>
    <server name="server-one" group="application-group" auto-start="true">
        <jvm name="default"/>
        <socket-bindings port-offset="100"/>
    </server>
    <server name="server-two" group="application-group" auto-start="true">
        <jvm name="default"/>
        <socket-bindings port-offset="200"/>
    </server>
</servers>

The “balancer” server is running on the default Port (8080) therefore you can access the backend Web application using this port:

$ curl http://localhost:8080/myapp

Hello there!

Configuring a static list of Proxies

In the second configuration, we will disable multicast to detect the list of Front End proxies. Instead, we will provide a static Proxy List of load balancers. The following changes are required:

On the Load Balancer server, we will disable the advertising by setting the advertise-frequency
attribute to zero. Also, we will unset the advertise-socket-binding attribute:

/profile=load-balancer/subsystem=undertow/configuration=filter/mod-cluster=load-balancer:write-attribute(name=advertise-frequency,value=0)

/profile=load-balancer/subsystem=undertow/configuration=filter/mod-cluster=load-balancer:undefine-attribute(name=advertise-socket-binding)

Then, on the cluster nodes we need to add as proxies the list of Front End WildFly Server:

/socket-binding-group=ha-sockets/remote-destination-outbound-socket-binding=proxy1:add(host=localhost, port=8090)

/profile=ha/subsystem=modcluster/proxy=default:write-attribute(name=proxies,value=[proxy1])

As you can see, we have set as remote destination the mod_cluster management Port which is by default 8090:

<socket-binding-group name="load-balancer-sockets" default-interface="public">
    <socket-binding name="http" port="${jboss.http.port:8080}"/>
    <socket-binding name="https" port="${jboss.https.port:8443}"/>
    <socket-binding name="mcmp-management" interface="private" port="${jboss.mcmp.port:8090}"/>
    <socket-binding name="modcluster" interface="private" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
</socket-binding-group>

As for our first example, can access the backend Web application using the load balancer’s port:

$ curl http://localhost:8080/myapp

Hello there!

Configuring Undertow as static load balancer (reverse proxy)

To configure undertow as static load balancer, you need to configure a proxy handler in its subsystem. This requires to complete a set of steps in your server that will act as static load balancer:

  • Add a reverse proxy handler
  • Define the outbound socket bindings for each remote host
  • Add each remote host to the reverse proxy handler
  • Add the reverse proxy location

We have added a tutorial to discuss more in detail this configuration: Configuring WildFly as Reverse Proxy

Load Balancer configuration for old WildFly Servers

Please note, if you are using a version prior to WildFly 10.1, there is no out of the box Undertow filter which connects to Mod Cluster via the advertising mechanism. You will need to manually add the Undertow filter, which will use mod_cluster’s advertise ports (port=23364, multicast-address=224.0.1.105). Here is the batch script we will need to execute on the WildFly front end CLI:

batch
/subsystem=undertow/configuration=filter/mod-cluster=modcluster:add(management-socket-
binding=http,advertise-socket-binding=modcluster)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=modcluster:add
# The following is needed only if you are not running an ha profile !**
/socket-binding-group=standard-sockets/socket-binding=modcluster:add(port=23364,
multicast-address=224.0.1.105)
run-batch

Now reload your configuration for the changes to take effect.

[standalone@localhost:9990/] reload

As a result, the modcluster filter has been added to the default-host server. Congratulations! You have just configured a WildFly server to act as Load Balancer!

How do I change multicast address of JBoss cluster ?

There are different options to change the default multicast address depending on the version of JBoss / WildFly which you are using.

Changing the multicast address in WildFly

In WildFly the System Property which controls the default multicast address is jboss.default.multicast.address and defaults to 230.0.0.4

<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>

Therefore, you can change it as follows:

$ ./standalone.sh -Djboss.default.multicast.address=230.0.0.8

You can also use the “-u <value>” as a shortcut to the system property:

$ ./standalone.sh -u 230.0.0.8

Please note that when using the jboss.socket.binding.port-offset it has no effect on multicast ports. Therefore if you need so define a server offset that will have no impact on the multicast address.

Changing the multicast address in JBoss 5

The -u (or --udp) command line switch may be used to control the multicast address used by the JGroups channels opened by all standard JBoss Enterprise Application Platform services.

$ ./run.sh -u 230.1.2.3 -g QAPartition -b 192.168.1.100 -c production

Clustering EJB 3 with JBoss AS

Please note: This tutorial has been written for JBoss EAP 5/6 and early versions of WildFly.

The current version of WildFly or JBoss EAP 7 no longer requires you to enable the clustering behavior. By default, if the server is started using the "ha" or "full-ha", the state of SFSBs will be replicated automatically and Stateless Session Beans will be load balanced.

If you want to disable this behavior for Stateful Session Beans check this tutorial: How to disable passivation for SFSB in WildFly

Clustering EJB 3 with JBoss EAP 6, JBoss EAP 5 and early WildFly versions

To cluster a stateless session bean in EJB 3 all you need to do is to annotate the bean class withe the @Cluster annotation. You can pass in the load balance policy and cluster partition as parameters to the annotation.

The default load balance policy is org.jboss.ha.framework.interfaces.RandomRobin and the default cluster is DefaultPartition.

Clustering a Stateless Session Bean

As we said earlier all you have to do is tagging the EJB with the @Stateless annotation before the class declaration. Here’s a sample:

package com.sample;

import java.io.Serializable;
import javax.ejb.*;
import org.jboss.annotation.ejb.Clustered;

@Stateless
@Clustered
public class HelloBean implements HelloBeanItf {
  int counter = 0;

  public String doSomething() {
     return "HelloBean Called"; 
  }

}

and here’s the interface:  

package com.sample;

public interface HelloBeanItf {

   public String doSomething(); 

}

Not very difficult, isn’t it ?? 

Clustering Stateful Session Beans

To cluster stateful session beans in EJB 3.0, you need to tag the bean implementation class with the @Cluster annotation, just as we did with the EJB 3.0 stateless session bean earlier.

@Stateful
@Clustered


public class HelloBean implements HelloBeanItf,Serializable {
int counter = 0;
  public void doSomething() {

     counter++;
     System.out.println("Value of counter is " + counter);

  }

So who takes care of session state replication ? the answer is JBoss Cache. JBoss Cache provides distributed cache and state replication services for the JBoss cluster. A JBoss cluster can have multiple JBoss Cache MBeans (known as the TreeCache MBean), one for HTTP session replication, one for stateful session beans, one for cached entity beans, etc.

The related MBean service is defined in the ejb3-clustered-sfsbcache-service.xml file in the deploy directory. The contents of the file are as follows.

<server>
   <mbean code="org.jboss.ejb3.cache.tree.PassivationTreeCache"
 name="jboss.cache:service=EJB3SFSBClusteredCache">
 
       <attribute name="IsolationLevel">READ_UNCOMMITTED</attribute>
       <attribute name="CacheMode">REPL_SYNC</attribute>
       <attribute name="ClusterName">SFSB-Cache</attribute>
       <attribute name="ClusterConfig">
           ... ...
       </attribute>

       <attribute name="SyncReplTimeout">10000</attribute>
       <attribute name="LockAcquisitionTimeout">15000</attribute>

       <attribute name="EvictionPolicyClass">org.jboss.ejb3.cache.tree.StatefulEvictionPolicy</attribute>

       <attribute name="EvictionPolicyConfig">
          <config>
             <attribute name="wakeUpIntervalSeconds">1</attribute>
            <name>statefulClustered</name>
            <region name="/_default_">
               <attribute name="maxNodes">1000000</attribute>
               <attribute name="timeToIdleSeconds">300</attribute>
            </region>
         </config>
      </attribute>

      <attribute name="CacheLoaderFetchPersistentState">false</attribute>
      <attribute name="CacheLoaderFetchTransientState">true</attribute>
      <attribute name="FetchStateOnStartup">true</attribute>
      <attribute name="CacheLoaderClass">org.jboss.ejb3.cache.tree.StatefulCacheLoader</attribute>
      <attribute name="CacheLoaderConfig">location=statefulClustered</attribute>
   </mbean>
</server>

JBoss AS 6 and 7 don’t use any more JBoss Cache for providing distributed and replicated cache. The new provider is Infinispan which exposes a data grid platform and exposes and a JSR-107 (JCACHE) compatible Cache interface

A local JBoss EJB client

A local client can be a simple jsp:

<%
   public void jspInit () {

   try {

     HelloBeanItf ejb = null;
     InitialContext ctx = new InitialContext();

     ejb = (HelloBeanItf)

       ctx.lookup("cluster/HelloBean/local");

   }

   catch (Exception exc) { 
      exc.printStackTrace();
   }
}

ejb.doSomething();
%>

Remember that in EJB 3, the JNDI of the EJB by default is made up of : 
package name( ear/jar) / Bean Name / local or remote

If you fail to find your EJB in JNDI tree take a look at the JNDI tree from the JMX console and you’ll solve the naming problem (There’s an how to in this site here : JBoss howtos )

How can a remote client access a clustered EJB ?

When using remote client the first thing you think about is JNDI.

How does JNDI works in a cluster ? JBoss service maintains a cluster-wide context tree called HA-JNDI (High Availability JNDI). The cluster wide tree is always available as long as there is one node left in the cluster.

In the previous JSP , new InitialContext(), will be bound to a local-only, non-cluster-wide JNDI Context (this is actually basic JNDI). So, all EJB homes and such will not be bound to the cluster-wide JNDI Context, but rather, each home will be bound into the local JNDI.


When a remote client does a lookup through HA-JNDI, HA-JNDI will delegate to the local JNDI Context when it cannot find the object within the global cluster-wide Context.

How the remote client is aware of the HA-JNDI cluster ? You can pass a list of JNDI servers (i.e., the nodes in the HA-JNDI cluster) to the java.naming.provider.url JNDI setting in the jndi.properties file. Each server node is identified by its IP address and the JNDI port number. The server nodes are separated by commas

Example:

java.naming.provider.url=server1:1100,server2:1100,server3:1100,server4:1100

you can pass the list of JNDI servers to the InitialContext also like here:

import java.util.Properties;

import javax.naming.Context;
import javax.naming.InitialContext;

public class Client {

  public static void main(String[] args) throws Exception {


     Properties p = new Properties(); 
     p.put(Context.INITIAL_CONTEXT_FACTORY,"org.jnp.interfaces.NamingContextFactory"); 
     p.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces"); 
     p.put(Context.PROVIDER_URL, "192.168.0.1:1100,192.168.0.2:1100");

     InitialContext ctx = new InitialContext(p);
 
     HelloBeanItf ejb = 
       (HelloBeanItf) ctx.lookup("cluster/HelloBean/remote");
 
     for (int ii=0;ii<10;ii++) {
        ejb.doSomething(); 
     }
     System.out.println("Terminated");
   }
}

Before running the sample remeber to put the @Remote annotation in the Bean class, other wise you will try to contact an EJB local interface from remote, and this is not allowed!

How to configure Load Balancing Groups in WildFly

In this tutorial we will learn how to configure Load Balancing Groups with mod_cluster so that requests for your applications are not spread randomly across the full WildFly cluster. Application requests will be sent first to other nodes available in a given Balancing Group before it is sent to another Load Balancing Group.

Our starting point is a domain server configuration which is made up of 4 servers:

  • server-one and server-two in the “main-server-group
  • server-three and server-four in the “other-server-group
    <servers>
        <server name="server-one" group="main-server-group"/>
        <server name="server-two" group="main-server-group" auto-start="true">
            <socket-bindings port-offset="150"/>
        </server>
        <server name="server-three" group="other-server-group" auto-start="true">
            <socket-bindings port-offset="250"/>
        </server>
        <server name="server-four" group="other-server-group" auto-start="true">
            <socket-bindings port-offset="350"/>
        </server>
    </servers>

Our server groups are bound to the “ha” profile configuration:

    <server-groups>
        <server-group name="main-server-group" profile="ha">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="full-sockets"/>
        </server-group>
        <server-group name="other-server-group" profile="ha">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="full-ha-sockets"/>
        </server-group>
    </server-groups>

Now let’s set a System Property on each Server Group to specify the Load Balancing Group to whom they belong:

/server-group=main-server-group/system-property=modcluster-balancing-group:add(value=main-group)

/server-group=other-server-group/system-property=modcluster-balancing-group:add(value=other-group)

Now, for your profile, configure the load-balancing-group attribute for your mod_cluster proxy:

/profile=ha/subsystem=modcluster/proxy=default:write-attribute(name=load-balancing-group,value=${modcluster-balancing-group})

/host=master:reload

You should have the following configuration in the profile section of your domain.xml:

<subsystem xmlns="urn:jboss:domain:modcluster:5.0">
    <proxy name="default" advertise-socket="modcluster" listener="ajp" load-balancing-group="${modcluster-balancing-group}">
        <dynamic-load-provider>
            <load-metric type="cpu"/>
        </dynamic-load-provider>
    </proxy>
</subsystem>

Then, within the server-groups section of the domain.xml file:

    <server-groups>
        <server-group name="main-server-group" profile="full">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="full-sockets"/>
            <system-properties>
                <property name="modcluster-balancing-group" value="main-group"/>
            </system-properties>
        </server-group>
        <server-group name="other-server-group" profile="full-ha">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="full-ha-sockets"/>
            <system-properties>
                <property name="modcluster-balancing-group" value="other-group"/>
            </system-properties>
        </server-group>
    </server-groups>

What happens in practice? when a new request mod_cluster tries to intelligently determine the best node available for the request.

Let’s assume that the first request lands on the server-one, which belongs to the “main-server-group” and the Load Balancing Group named “main-group”. Now what happens if server-one fails or it’s shutdown? As mod_cluster is aware that there is another server in the same Load Balancing Group, then server-two will serve the next requests.

The advantage of using a Load Balancing Group is that we can make the load balancing of your application more deterministic. This can make your cluster more scalable, for example if you don’t want to replicate your HTTP Session across all 4 cluster members but just across your Load Balancing Group.

WildFly Configuring HTTP Session in a cluster

This tutorial discusses in detail how to configure HTTP Session Management in clustered Web applications, based on the new changes introduced in WildFly 17.

Web applications which are running in a cluster-aware configuration (“ha” or “full-ha” profile) can survive the lifespan of a single server by adding the element into the web.xml file. Example:

<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_4_0.xsd"
         version="4.0">
    <distributable/>
</web-app>

This is sufficient to achieve that your HTTP Session survives a server crash/restart, provided that at least one server is still available. You can further specialize the way HTTP Session is managed in the cluster using different strategies, depending on your WildFly version.

HTTP Session Management in WildFly 17 or newer

WildFly 17 introduced the distributable-web subsystem, which manages a set of session management profiles that encapsulate the configuration of a distributable session manager:

   <subsystem xmlns="urn:jboss:domain:distributable-web:2.0" default-session-management="default" default-single-sign-on-management="default">

You can check the default-session-management attribute from the CLI as follows:

 /subsystem=distributable-web:read-attribute(name=default-session-management)
{
    "outcome" => "success",
    "result" => "default"
}

The HTTP Session Management is handled, under the hoods,by Infinispan, therefore if you want to check its settings, you have to reach the infinispan-session-management attribute under the distributable-web subsystem. Example, for the “default” infinispan-session-management:

 /subsystem=distributable-web/infinispan-session-management=default:read-resource
{
    "outcome" => "success",
    "result" => {
        "cache" => undefined,
        "cache-container" => "web",
        "granularity" => "SESSION",
        "affinity" => {"primary-owner" => undefined}
    }
}

As you can see, there are several configurable attribute for the infinispan-session-management:

  • cache-container: This references the Infinispan cache-container into which session data will be stored.
  • cache: This references a cache for the related cache-container. If undefined, the default cache of the associated cache container will be used.
  • granularity: This defines the session manager mapping for the individual cache entries:
  • affinity: This attribute defines the affinity that a web request should have for a given server.

Configuring HTTP Session Granularity

The granularity attribute can have the following values:

  • SESSION: Stores all session attributes within a single cache entry. This is generally more expensive than ATTRIBUTE granularity, but preserves any cross-attribute object references.
  • ATTRIBUTE: Stores each session attribute within a separate cache entry. This is generally more efficient than SESSION granularity, but does not preserve any cross-attribute object references.

If your application does not share any object references between attributes, users are strongly advised to use ATTRIBUTE granularity. Using ATTRIBUTE granularity, each session attribute is stored in a separate cache entry. This means that a given request is only required to replicate/persist those attributes that were added/modified/removed/mutated in a given request. For read-heavy applications, this can dramatically reduce the replication/persistence payload per request.

Here is how you can set the granularity to “ATTRIBUTE” for the default session manager:

/subsystem=distributable-web/infinispan-session-management=default/:write-attribute(name=granularity,value=ATTRIBUTE)

Configuring HTTP Session Affinity

The affinity attribute defines the affinity that an HTTP request has for a given WildFly server. The affinity of the associated web session determines the algorithm for generating the route to be appended onto the session ID (within the JSESSIONID cookie, or when encoding URLs). Possible values are:

  • affinity=none: HTTP requests won’t have affinity to any particular node.
  • affinity=local: HTTP requests will have an affinity to the server that last handled a request for a given session. This is the standard sticky session behavior.
  • affinity=primary-owner: HTTP requests will have an affinity to the primary owner of a given session. This is the default setting. Behaves the same as affinity=local if the backing cache is not distributed nor replicated.

Adding a new Session Management Profile

You can define a new Session Management Profile as follows:

/subsystem=distributable-web/infinispan-session-management=custom-profile/:add(cache-container=web,granularity=ATTRIBUTE)

Now, in order to link the “custom-manager” Session Management Profile to your application, several options are available:

Include a WEB-INF/distributable.xml file in your application, which links your customer profile:

<?xml version="1.0" encoding="UTF-8"?>
<distributable-web xmlns="urn:jboss:distributable-web:1.0">
    <session-management name="custom-profile"/>
</distributable-web>

You can also link the Session Management Profile through the existing jboss-all.xml:

<?xml version="1.0" encoding="UTF-8"?>
<jboss xmlns="urn:jboss:1.0">
    <distributable-web xmlns="urn:jboss:distributable-web:1.0">
        <session-management name="custom-profile"/>
    </distributable-web>
</jboss>

Defining Session Management Profile at application level

It is also possible to use deployment-specific settings for your session management profile. This can be done by adding an infinispan-session-management element into the /WEB-INF/distributable-web.xml. Example:

<?xml version="1.0" encoding="UTF-8"?>
<distributable-web xmlns="urn:jboss:distributable-web:1.0">
  <infinispan-session-management cache-container="web" cache="demo-cache" granularity="ATTRIBUTE">
            <local-affinity/>
  </infinispan-session-management>
</distributable-web>

As an alternative, also the file META-INF/jboss-all.xml can contain deployment specific settings for your session profile:

<?xml version="1.0" encoding="UTF-8"?>
<jboss xmlns="urn:jboss:1.0">
    <distributable-web xmlns="urn:jboss:distributable-web:2.0">
      <infinispan-session-management cache-container="web" granularity="SESSION">
        <primary-owner-affinity/>
      </infinispan-session-management>
    </distributable-web>
</jboss>

Using jboss-web.xml to manage max-active-sessions

Prior to WildFly 17, the file jboss-web.xml contained the core settings for configuring HTTP Session replication. Here is the basic structure of it:

<jboss-web>
  ...
  <max-active-sessions>...</max-active-sessions>
  ...
  <replication-config>
    <replication-granularity>...</replication-granularity>
    <cache-name>...</cache-name>
  </replication-config>
  ...
</jboss-web>

Since WildFly 17, the replication-config section has been deprecated. You can still use max-active-sessions to impose a limit to the number of currently active sessions. Please notice that this limit works differently, depending if you are running an application in a cluster or not (that is if you are using in web.xml).

  • Clustered Web applications: if you reach the max-active-sessions and a new request arrives, the request is accepted and an old session is passivated to disk using a Least Recently Used (LRU) algorithm
  • Not Clustered Web applications: when exceeding the max-active-sessions limit, a new request will fail with an IllegalStateException.

Configuring HTTP Session replication before WildFly 17

Prior to WildFly 17, you can rely on jboss.web.xml to configure the replication configuration for HTTP Sessions. For example, the following jboss-web.xml snippet shows how to set the replication-granularity to SESSION for your Web application:

<jboss-web xmlns="http://www.jboss.com/xml/ns/javaee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.jboss.com/xml/ns/javaee http://www.jboss.org/j2ee/schema/jboss-web_10_0.xsd">
   <replication-config>
      <replication-granularity>SESSION</replication-granularity>
    </replication-config>
</jboss-web>

Older WildFly versions could also configure the replication-trigger of your HTTP Session as in the following example:

<jboss-web>
  <replication-config>
    <replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger>
    <replication-granularity>SESSION</replication-granularity>
  </replication-config>
</jboss-web>

There are 4 possible levels of replication of your session data: each one triggers the session replication between nodes in a different way: 

Level Description Speed
SET This is the best option for performance. The session is replicated only if an attributed is explicity modified with setAttribute method. good
SET_AND_GET With this option any attributed that is get/set is marked as dirty even if it’s not written back to the session. This leads to a significant performance degradation. slow
SET_AND_NON_
PRIMITIVE_GET
This is the default option. It works the same as SET_AND_GET except for primitive system types (String, Integer, Long). Since they are immutable objects they are not replicated when a get is issued. average
ACCESS This is the most conservative option. It causes the session to be marked as dirty whenever it is accessed. Since a the session is accessed during each HTTP request, it will be replicated with each request. Note that use of this option can have a significant performance impact, so use it with caution v.slow

In the following table you can see a set of commands and whether they make the session dirty.

Command SET SET_AND_NP_GET SET_AND_GET ACCESS
HttpSession session; NO NO NO YES
String s = (String)session.getAttribute(“x”); NO NO YES YES
Person p = (String)session.getAttribute(“p”); NO YES YES YES
Person p = (String)session.getAttribute(“p”);
session.setAttribute(p);
YES YES YES YES

Configuring Ranked Load Balancing in WildFly

One of the new features included in WildFly 18 is the ability to load balance requests using a ranked order of preference.

The default session affinity algorithm in a WilFly cluster is set by the “affinity” attribute:

/subsystem=undertow/configuration=filter/mod-cluster=load-balancer:read-resource()
{
    "outcome" => "success",
    "result" => {
        "advertise-frequency" => 10000,
         . . . .
        "affinity" => {"single" => undefined},
        "balancer" => undefined
    }
}

This means that, out of the box, web requests have an affinity for the member that last handled a given session. This option corresponds to traditional sticky session behavior.

By using ranked session affinity means that the WildFly is able to annotate the JSESSIONID with multiple routes, ranked in order of preference. Thus, if the primary owner of a given session is inactive, the load balancer can attempt to route the request to the next route in the list. This ensures that requests will be directed to the next best worker in the event that the primary owner is inactive, and prevents requests from “spraying” across the cluster.

The load balancer must be explicitly configured to enable support for parsing ranked routing; initially supporting Undertow-based load balancer.

When using WildFly as a load balancer ranked routing can be enabled with the following CLI command:

/subsystem=undertow/configuration=filter/mod-cluster=load-balancer/affinity=ranked:add

In this case, Web requests will have an affinity for the first available node in a list typically comprised of: primary owner, backup nodes, local node (if not a primary nor backup owner).

Configuring High Availability with WildFly

High Availability in WildFly serves the purpose to guarantee the availability of Java EE application. As the number of request for an application increase, there are two potential issues:

  • Failure of the application, if the only node where it runs crashes.
  • Delays in serving requests, if the server is unable to handle the load.

In order to prevent these issues, WildFly supports two core features to ensure high availability of Java EE applications:

  • failover: which allows a client to continue running the application, even in case of node failure, without losing data
  • load balancing: which allows to server to provide timely responses, even in the presence of high-volumes of load

Handling WildFly failover

Failover allows a client interacting with a Java EE application to have uninterrupted access to that application, even in the presence of node failures.

For example, consider a Java EE application which makes use of the following features:

  • HTTP Session aware Servlets to provide user interaction
  • Stateful EJBs to perform state-dependent business operations
  • JPA Entity to store critical data in a persistent store
  • SSO login to the application

If the application makes use of WildFly’s fail-over services, a client interacting with an instance of that application will not be interrupted even when the node on which that instance executes crashes.

Behind the scenes, WildFly makes sure that all of the user data that the application make use of are available at other nodes in the cluster, so that when a failure occurs and the client is redirected to that new node for continuation of processing, the user’s data is available and processing can continue.

The following subsystems are involved in implementing highly available applications:

  • jgroups

    JGroups is a toolkit for the underlying communication between nodes. Two main communication stacks are available in EAP: UDP (default) or TCP (no multicast).

  • infinispan

    Infinispan is used for data caching and object replication. In WildFly, Infinispan ships with 4 preconfigured caches (cluster, web, SFSB and hibernate).

  • modcluster

    mod_cluster is an httpd-based load balancer. Like mod_jk and mod_proxy, mod_cluster uses a communication channel to forward requests from httpd to one of a set of application server nodes.

  • singleton

    The singleton subsystem can be used for the deployment of applications as highly-available singletons.

  • undertow

    Undertow is the new web server project integrated in WildFly. It is a flexible and high performing Java web server providing NIO based blocking and non-blocking APIs.

In order to allow high availability of an application using these features, it is enough to start the application server using a profile which provides high availability. For example, if you need high availability of EJB or Web applications:

$ ./standalone.sh -c standalone-ha.xml

On the other hand, if you need high availability of EJB, Web and JMS applications, you have to use the full-ha profile:

$ ./standalone.sh -c standalone-full-ha.xml

Check out these tutorials which are about Clustering EJB , Web and JMS applications:

EJB Applications: How to make all your EJB clusterable in JBoss/WildFly

JMS Applications: JMS Clustering in WildFly and JBoss EAP

Handling WildFly load balancing

Load balancing allows you to distribute the workload across multiple nodes, in our case, WildFly cluster nodes. This technique is used for optimizing resources, minimizing the response time, and maximizing the application throughput. To achieve load balancing, we need a component which fronts our WildFly nodes, and distributes the workload across them. A common pattern of distributing the workload is to forward client requests in a round-robin manner, so that every node serves the same number of requests.

Some options to achieve load balancing include: