Monitoring WildFly using VisualVM

In this updated tutorial we will learn how to connect the latest version of WildFly application server using VisualVM monitoring tool.

VisualVM is a free tool to monitor and profile Java application. In the past, VisualVM used to be shipped with Oracle JDK 6~8 as Java VisualVM. It has been discontinued in Oracle JDK 9. Today, VisualVM is distributed as a standalone tool and bundled with the GraalVM. Both are the same bits with the same features. Standalone tool runs on any compatible JDK, bundled tool is configured to run using the host GraalVM.

If you opt to download VisualVM as standalone application, you can grab a copy of it from here: https://visualvm.github.io/download.html

Then, unzip the downloaded archive. The archive already contains the top-level visualvm directory. Start VisualVM by invoking the binary appropriate for your OS, which is located in the bin folder:

$ ./visualvm

The following sections provide instructions for using VisualVM to connect to a local or remote WildFly JVM. As for JConsole, we will at first check out how to connect to a local JVM process and then to a remote server.

Connecting to a Local WildFly JVM Using VisualVM

Before we start WildFly, we need to add to WildFly modules the JFluid Server API which allows starting the full instrumentation profiling, calls to which are injected into the target application bytecodes when they are instrumented.

The required library, named jfluid-server-15.jar, is included into VisualVM distribution in the directory path $VISUALVM_HOME/visualvm/lib/

First, we need to make the JFluid API available as System Packages. This requires to include its top packages in the JBOSS_MODULES_SYSTEM_PKGS environment variable. For standalone servers, open the file standalone.conf and add the package ‘org.graalvm.visualvm’ right after byteman’s packages:

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.graalvm.visualvm"
fi

For domain mode, locate the system-properties element in domain.xml file and add a property named jboss.modules.system.pkgs with a value of org.graalvm.visualvm to the existing system properties. For example:

<system-properties>
        <property name="jboss.modules.system.pkgs" value="org.graalvm.visualvm"/>
</system-properties>

Then, we need to provide the JFluid jar files to WildFly Runtime. A handy option is to make this library available as part of a global directory which is scanned at start by WildFly. Let’s see how to do that. Connect to WildFly CLI:

$ ./jboss-cli.sh -c

Then execute the following command (replace the path with the actual path where VisualVM has been installed)

/subsystem=ee/global-directory=profiler:add(path=/home/jboss/visualvm_206/visualvm/lib)

The following configuration will be added to your WildFly server:

<subsystem xmlns="urn:jboss:domain:ee:5.0">
    <global-directories>
        <directory name="my-common-libs" path="/home/jboss/visualvm_206/visualvm/lib"/>
    </global-directories>

To connect to a WildFly JVM running on the same machine as VisualVM:

  • Open VisualVM, and find the `Applications` panel on the left side of the VisualVM window.
  • Under `Local`, double-click the WildFly JVM process that you want to monitor.

For a standalone WildFly server, there is one WildFly JVM process.

A WildFly managed domain host has multiple JVM processes you can connect to: a *Host controller* JVM process, a *Process controller* JVM process, and a JVM process for each WildFly *Server* on the host. You can find out the list of active Java processes using the jps command as follows:

$ jps -lv

5469 /home/jboss/wildfly-22.0.0.Final/jboss-modules.jar -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m 

Right-click on the WildFly server connection and choose Open. Once connected, the following tabs will display in the Main window:

From each tab, the following options will be available:

  • `Overview`: This tab shows generic server info like JVM arguments and system properties.
  • `Monitor`: This tab reveals a real-time set of graphs which collect data for the CPU, memory utilization, classloading, and threads used.
  • `Threads`: This tab provides a detailed view of all threads running in the JVM along with their state (Running, Sleeping, Wait, and Monitor).
  • `Profiler`: This tab enables you to instrument a profiling session of a local application.
  • `Sampler`: This tab enables to capture Heap snapshots periodically and display memory/cpu statistics per class.

The standard monitoring options are straightforward to use and don’t need an accurate guidance. In order to pinpoint performance issues, you should rather focus on the Profiler and Sampler options.

Troubleshooting VisualVM Connection issues

If your connection to the WildFly process is hanging on and you are sure the connection should work (i.e. you double checked all the standard things like firewalls, network configuration etc.) the RMI system property java.rmi.server.hostname is something to try:

$ ./standalone.sh -Djava.rmi.server.hostname=localhost

Besides it, it’s worth checking the Network options of VisualVM. Verify that your Proxy settings are set to “No Proxy” or, as an alternative, that the target host is in the list of excluded proxy hosts.

JBoss and WildFly latest version

JBoss and WildFly latest versions (updated December 2020)

Product Latest Version Type Download link
WildFly Application Server 23.0.0 Community https://www.wildfly.org/downloads/
JBoss EAP 7.3.5 Subscription Required https://access.redhat.com/jbossnetwork/restricted/softwareDownload.html?softwareId=90971
JBoss EAP XP 2.0.0 Subscription Required https://access.redhat.com/jbossnetwork/restricted/softwareDownload.html?softwareId=90881

Getting started with WildFly

Let’s get started with WildFly! WildFly is a Java middleware product also known as application server. The word “application server” has been coined in relation to Java Enterprise application; you can think of it as it’s a piece of Java software where your application can be provisioned using the services provided by the application server.

Installing WildFly

The pre-requisite to the Application Server installation is that you have available a JDK on your machine.

It is recommended to use a JDK 9 or higher to start WildFly. JDK can be either downloaded from Oracle site at http://www.oracle.com/technetwork/java/javase/downloads/index.html or you can use an open source implementation of it called OpenJDK http://openjdk.java.net/

Once installed the JDK, you have to set the JAVA_HOME environment variable accordingly.

Done with JDK installation, let’s move to the application server. WildFly can be downloaded from http://www.wildfly.org by following the Downloads link in the home page.

The latest version of WildFly is: 27.0.0.Alpha4

Once downloaded, extract the archive to a folder and you are done with the installation.

$ unzip wildfly-27.0.0.Alpha4.zip

The application server ships with two server modes: standalone and domain mode. The difference between the two modes is not about the capabilities available but is related to the management of the application server: in particular, the domain mode is used when you run several instances of WildFly and you want a single point where you can manage servers and their configuration.

In order to start WildFly using the default configuration in “standalone” mode, change the directory to $JBOSS_HOME/bin and issue:

$ ./standalone.sh

To start the application server using the default configuration in “domain” mode, change directory to $JBOSS_HOME /bin and execute:

$ ./domain.sh

When starting in standalone mode, you should find in your console something like this, at the end of start up process:

15:27:20,698 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 23.0.2.Final (WildFly Core 14.0.0.Final) started in 3303ms - Started 314 of 580 services (370 services are lazy, passive or on-demand)
15:27:20,700 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening o http://127.0.0.1:9990/management
15:27:20,700 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990

You can verify that the server is reachable from the network by simply pointing your browser to the application server’s welcome page, which is reachable by default at the following address:

http://localhost:8080

Your first task: Create a WildFly Administrator

If you want to manage the application server configuration using its management instruments, you need to create a management user.

In order to create a new user, just execute the add-user.sh/add-user.bat, which is located in the bin folder of the application server’s home.

  1. Select the default option “a” to add a Management user. This user is added to the ManagementRealm. Therefore it is authorized to perform management operations using the web-based Admin Console or the CLI (command-line interface). The other choice, b, adds a user to the ApplicationRealm, That realm is provided for use with applications.
  2. Enter the desired username and password.
  3. When prompted, enter the username and password. You will be prompted to confirm the password.
  4. Enter group information.
  5. Add the group or groups to which the user belongs. If the user belongs to multiple groups, enter a comma-separated list. Leave it blank if you do not want the user to belong to any groups.
  6. Review the information and confirm. If you are satisfied, type yes.

Here is a transcript which summarizes these steps:

$ ./add-user.sh 

What type of user do you wish to add? 
 a) Management User (mgmt-users.properties) 
 b) Application User (application-users.properties)
(a): a

Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : admin123
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
 - The password should be different from the username
 - The password should not be one of the following restricted values {root, admin, administrator}
 - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
Password : 
Re-enter Password : 
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]: 
About to add user 'admin123' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'admin123' to file '/home/francesco/jboss/wildfly-23.0.2.Final/standalone/configuration/mgmt-users.properties'
Added user 'admin123' to file '/home/francesco/jboss/wildfly-23.0.2.Final/domain/configuration/mgmt-users.properties'
Added user 'admin123' with groups  to file '/home/francesco/jboss/wildfly-23.0.2.Final/standalone/configuration/mgmt-groups.properties'
Added user 'admin123' with groups  to file '/home/francesco/jboss/wildfly-23.0.2.Final/domain/configuration/mgmt-groups.properties'
Is this new user going to be used for one AS process to connect to another AS process? 
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? yes
To represent the user add the following to the server-identities definition <secret value="UGFzc3dvcmQxMjM=" />

Now that we have got started with WildFly, let’s continue learning: Getting to know WildFly folder structure

Sharing HTTP Session between Web applications in an EAR

One of the new features of WildFly 9 is the ability to share the HTTP Session between applications which are part of the same Enterprise Archive. (This feature is described in https://issues.jboss.org/browse/WFLY-1891 )

Undertow allows you to share sessions between wars in an ear, if it is explicitly configured to do so. Note that if you use this feature your applications may not be portable, as this is not a standard Servlet feature, although it’s available in other vendors like Websphere and Oracle Weblogic.

Let’s see an example application which consists of two Web applications: webapp1 and webapp2, bundled into one EAR archive:

As simple test, both our webapplications will include an index.jsp page which adds one random attribute to the session and dumps the content of the HttpSession:

<%
session.setAttribute(java.util.UUID.randomUUID().toString(), new java.util.Date().toString());

 
java.util.Enumeration enames = session.getAttributeNames();
while (enames.hasMoreElements()) {
   String key = (String) enames.nextElement();
   String value = "" + session.getAttribute(key);
   out.println(key + " - " + value); 
}

%>

HttpSession sharing is not enabled by default, we need to include a jboss-all.xml file in the META-INF folder of the EAR file, including a shared-session-config element within it:

<jboss umlns="urn:jboss:1.0">
    <shared-session-config xmlns="urn:jboss:shared-session-config:2.0">
        <session-config>
            <cookie-config>
                <path>/</path>
            </cookie-config>
        </session-config>
    </shared-session-config>
</jboss>

Now if you try to switch from one Web application to another you will see that the HttpSession is correctly maintained across the two Web applications.

If you want to check the full list of attributes you can apply, check the XSD file: https://github.com/wildfly/wildfly/blob/main/undertow/src/main/resources/schema/shared-session-config_2_0.xsd

Create a Singleton in a Cluster with WildFly

Using a Singleton Service in a cluster is not something completely new concept. In JBoss 6 you could use a SingletonService to decorate a Service such that the service will only ever be started on one node in a cluster at any given time. WildFly 10 brings again to life this service with a fully configurable subsystem which allows fine tuning of the Singleton Service:

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
    <singleton-policies default="default">
        <singleton-policy name="default" cache-container="server">
            <simple-election-policy/>
        </singleton-policy>
    </singleton-policies>
</subsystem>

What is configurable in the subsystem is first of all the policy. Out of the box the singleton subsystem uses the default policy which creates a circular list of cluster members, starting from the oldest, which are in turn elected to host the singleton application. If you prefer rather a random policy, you can at any time vary it from the CLI with:

/subsystem=singleton/singleton-policy=foo/election-policy=random:add()

The other important thing is the cache-container which must reference a valid cache from the Infinispan subsystem. If you are using the standard configuration, the default “server” cache will be used.

Creating a Singleton application

In order to activate the Singleton Service you need to provide an XML configuration file named singleton-deployment.xml to be placed in the META-INF folder of your JAR/WAR archive:

src
└── main
    ├── resources
    │   ├── META-INF
    │   │   └── singleton-deployment.xml
    └── webapp
        └── index.jsp

 Here is a minimalist singleton-deployment.xml:

<singleton-deployment xmlns="urn:jboss:singleton-deployment:1.0"/>

Once that the XML file is picked up, the application will be deployed in your cluster just on the elected provider. You can see it from your logs that the application web.war deployed in the cluster made of server-one and server-two has been activated on server-two, which will act as master:

[Server:server-two] 12:17:35,310 INFO  [org.wildfly.clustering.server] (notification-thread--p4-t1) WFLYCLSV0003: master:server-two elected as the singleton provider of the jboss.deployment.unit."web.war".FIRST_MODULE_USE service
[Server:server-two] 12:17:35,332 INFO  [org.wildfly.clustering.server] (notification-thread--p4-t1) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."web.war".FIRST_MODULE_USE service

You can test your application which is not just available on server-two (say on port 8180):

$ curl http://localhost:8180/web/index.jsp
Hello world!

Now, as we stop server-two, a new master will be elected:

/host=master/server-config=server-two:stop

As a result, the application is available now on server-one (offset 0):

You can test your application which is not just available on server-two (say on port 8180):

$ curl http://localhost:8080/web/index.jsp
Hello world!

Practical usages of the Singleton Service

One practical usage of the Singleton Service is the EJB Timer which can be easily configured to run on a single node in the cluster so that you get Timer notifications just from a node in a cluster, which however provides High Availability of the timer, in case the node shuts down.

For example, see the following Timer EJB taken from: https://github.com/wfink/jboss-eap-playground/tree/master/eap7ClusterHASingletonDeployment/ejb

@Stateless
public class SimpleBean implements SimpleRemote {
    private static final Logger log = Logger.getLogger(SimpleBean.class);

    @Resource
    SessionContext context;

    private String getJBossNodeName() {
        return System.getProperty("jboss.node.name");
    }

    @Schedule(hour="*", minute="*", second="*/10", persistent=false)
    public void showRunning() {
        log.infof("Timer is active @%s", getJBossNodeName());
    }

    @Override
    public void invoke(String text) {
        log.infof("Bean invocation @%s: %s", getJBossNodeName(), text);
    }
}

All you need to turn this Timer EJB in a Singleton Timer is the singleton-deployment.xml to be placed in the META-INF folder of your JAR and you’re done.

WildFly 10.1 new features

WildFly 10.1 has been released. Let’s see which are the most significant enhancements introduced in the application server.

New load-balancing profile

One of the new features available since WildFly 9, is the ability to act as a mod_cluster based front­end for your back­end applications. This will remove the need to use a native Web server like Apache (and mod_cluster libs installed on it) as load balancer to a cluster of WildFly servers.

Here is a sample view of a cluster of WildFly backend servers fronted by a WildFly frontend server configured to route request using the Mod_cluster Management Protocol (MCMP):

With configurations prior to WildFly 10.1 you needed to create an Undertow filter in your configuration which filters traffic to the advertise socket binding of mod_cluster. Now the application server ships out of the box with a configuration named standalone-load-balancer.xml which is a minimal WildFly Web server configuration that can be used as front end, instead of Apache Web server:

<subsystem xmlns="urn:jboss:domain:undertow:3.1">
    <buffer-cache name="default"/>
    <server name="default-server">
        <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
        <http-listener name="management" socket-binding="mcmp-management" enable-http2="true"/>
        <host name="default-host" alias="localhost">
            <filter-ref name="load-balancer"/>
        </host>
    </server>
    <servlet-container name="default"/>
    <filters>
        <mod-cluster name="load-balancer" management-socket-binding="mcmp-management" advertise-socket-binding="modcluster" enable-http2="true"/>
    </filters>
</subsystem>

If you are using Domain mode there is a corresponding load-balancer profile which can be used for the same purpose:

<profile name="load-balancer">
 . . .
</profile>

Built-in HTTP/2 capabilities

HTTP/2 is the new version of the HTTP /1.1 protocol. It improves the communication in many aspects: first of all it’s binary, instead of textual. Binary protocols are far more efficient to parse, more compact “on the wire”, and less error prone. Also HTTP/2 is fully multiplexed. Multiplexing addresses allows multiple request and response messages to be routed at the same time; it’s even possible to mix parts of one message with another on the wire.

HTTP/2 also requires a TLS security stack, therefore in order to use it you had to exploit a complex procedure in order to enable the application server’s JVM to use it.

This is not needed anymore as by default the application server ships with an https-listener that is capable of using HTTP/2:

<https-listener name="https" socket-binding="https" security-realm="ApplicationRealm" enable-http2="true"/>

A demo certificate has been installed and configured in the ApplicationRealm:

<security-realm name="ApplicationRealm">
        <server-identities>
            <ssl>
                <keystore path="application.keystore" relative-to="jboss.server.config.dir" keystore-password="password" alias="server" key-password="password" generate-self-signed-certificate-host="localhost"/>
            </ssl>
        </server-identities>
        <authentication>
            <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
            <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
        </authentication>
        <authorization>
            <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
        </authorization>
</security-realm>

You can try starting the application server and you will notice that the self-signed certificate has been created for you:

15:53:55,896 WARN  [org.jboss.as.domain.management.security] (default I/O-2) WFLYDM0113: Generated self signed certificate at /home/francesco/jboss/wildfly-10.1.0.Final/standalone/configuration/application.keystore. Please note that self signed certificates are not secure, and should only be used for testing purposes. Do not use this self signed certificate in production.
SHA-1 fingerprint of the generated key is 51:99:b6:42:d4:cc:41:bc:66:68:bd:5f:47:fe:e3:c7:da:8d:b0:ad
SHA-256 fingerprint of the generated key is 51:58:18:fb:16:33:8f:90:eb:31:d9:79:de:77:78:2b:c0:ed:56:af:b2:00:6a:20:e5:62:c5:ac:09:24:3c:f3

You can verify that the HTTP/2 protocol is active by pointing the browser to a secure port (ex. https://localhost:8443). The response is then “HTTP/2.0” something and Firefox inserts its own header called “X­Firefox­Spdy:” as shown in the screenshot below:

Enable Clustering on MS Azure

WildFly 10.1 now by default include support, at JGroups level, for Azure Cloud service. This is not included in the default profile of the server, but you will find a demo configuration in the JBOSS_HOME/docs/examples/configs/standalone-azure.xml. Here is the relevant configuration which instead of the default JGroups’ PING protocol has the azure.AZURE_PING protocol:

   <stacks>
        <stack name="udp">
            <transport type="UDP" socket-binding="jgroups-udp">
                <property name="ip_mcast">false</property>
            </transport>
            <protocol type="azure.AZURE_PING">
                <property name="storage_account_name">${jboss.jgroups.azure_ping.storage_account_name}</property>
                <property name="storage_access_key">${jboss.jgroups.azure_ping.storage_access_key}</property>
                <property name="container">${jboss.jgroups.azure_ping.container}</property>
            </protocol>
            <protocol type="MERGE3"/>
            <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
            <protocol type="FD"/>
            <protocol type="VERIFY_SUSPECT"/>
            <protocol type="pbcast.NAKACK2">
                <property name="use_mcast_xmit">false</property>
                <property name="use_mcast_xmit_req">false</property>
            </protocol>
            <protocol type="UNICAST3"/>
            <protocol type="pbcast.STABLE"/>
            <protocol type="pbcast.GMS"/>
            <protocol type="UFC"/>
            <protocol type="FRAG2"/>
        </stack>
        <stack name="tcp">
            <transport type="TCP" socket-binding="jgroups-tcp"/>
            <protocol type="azure.AZURE_PING">
                <property name="storage_account_name">${jboss.jgroups.azure_ping.storage_account_name}</property>
                <property name="storage_access_key">${jboss.jgroups.azure_ping.storage_access_key}</property>
                <property name="container">${jboss.jgroups.azure_ping.container}</property>
            </protocol>
            <protocol type="MERGE3"/>
            <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
            <protocol type="FD"/>
            <protocol type="VERIFY_SUSPECT"/>
            <protocol type="pbcast.NAKACK2"/>
            <protocol type="UNICAST3"/>
            <protocol type="pbcast.STABLE"/>
            <protocol type="pbcast.GMS"/>
            <protocol type="FRAG2"/>
        </stack>
    </stacks>

Simply start the application server including the correct values for the System Properties. ex:

$ ./standalone.sh -Djboss.jgroups.azure_ping.storage_account_name="A" -Djboss.jgroups.azure_ping.storage_access_key="B" -Djboss.jgroups.azure_ping.container="C"

Upgrading existing JGroups configurations

If you want to upgrade an existing WildFly configuration to use AZURE_PING, just execute the following batch script which removes at first the default MPING protocol and then adds the azure.AZURE_PING protocol to the TCP JGroups stack:

batch
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
/subsystem=jgroups/stack=tcp/protocol=MPING:remove
/subsystem=jgroups/stack=tcp/protocol=azure.AZURE_PING:add(add-index=0,properties=[storage_account_name=${jboss.jgroups.azure_ping.storage_account_name:},storage_access_key=${jboss.jgroups.azure_ping.storage_access_key:},container=${jboss.jgroups.azure_ping.container:}])
run-batch
/:reload