JBoss cheatsheet for Linux Administrators (2022)

This is my JBoss / WildFly cheatsheet I use for managing the application server on a Linux box. Enjoy it and welcome any contribution to the cheatsheet.

WildFly / JBoss CLI

How to execute commands in non-interactive way:

jboss-cli.sh --connect --command=":reload"

How to connect to a non default host/port

jboss-cli.sh --connect --controller=192.168.0.1:9990

How to execute script in a file

jboss-cli.sh --file=test.cli

How to run in GUI mode:

jboss-cli.sh --gui

How to run in Admin-only mode:

jboss-cli.sh --admin-only

How to shutdown and restart the server:

shutdown --restart=true

How to start the server in embedded mode:

embed-server --server-config=server-config.xml --std-out=echo

How to stop the embedded server:

[standalone@embedded /] stop-embedded-server

How to suspend a server with a timeout:

:suspend(timeout=60)

Resume a server:

:resume

WildFly Domain

Start all servers in a Host:

/host=master:start

Stop all servers in a Host:

/host=master:stop

Suspend all servers in a Domain:

:suspend-servers(timeout=60)

Suspend a single server in a managed domain.

/host=master/server-config=server-one:suspend(timeout=60)

Suspend all servers in a server group.

/server-group=main-server-group:suspend-servers(timeout=60)

JVM

Standalone mode

Set in standalone.conf:

JAVA_OPTS="-Xms64m -Xmx512m -XX:MaxMetaspaceSize=256m"

Domain mode

Set the default heap settings for a domain Host.

/host=HOST/jvm=production_jvm:add
(
heap-size=2048m,
max-heap-size=2048m,
max-permgen-size=512m,
stack-size=1024k,
jvm-options=["-XX:-UseParallelGC"]
)

Set the default heap settings for a domain host:

/host=HOST/jvm=default:write-attribute(name=heap-size,value="1024m")

Set the heap size for a server group:

/server-group=groupName/jvm=default:write-attribute(name=heap-size,value="1024m")

Set the heap size for a single server:

/host=HOST/server-config=server-one/jvm=default:write-attribute(name=heap-size,value="1024m")

Set a server to use the default JVM settings for the domain.

/host=HOST/server-config=server-one/jvm=production_jvm:add()

Deploy

Deploy an application to a standalone server.

deploy /path/to/app/myapp.war

Deploy an application to a Server Group:.

deploy --server-groups=main-server-group /path/to/app/app.war

Batch

[standalone@localhost:9990 /] batch
[standalone@localhost:9990 / #] /subsystem=datasources/data-source="java\:\/H2DS":enable
[standalone@localhost:9990 / #] /subsystem=messaging-activemq/server=default/jms-queue=newQueue:add
You can execute a batch using the run-batch command:
[standalone@localhost:9990 / #] run-batch

Variables

How to set a variable in CLI:

set s1=/host=master/server=server-one
$s1/subsystem=datasources/data-source=ExampleDS:test-connection-in-pool

Alias

How to create an alias in CLI:

alias create='/subsystem=ejb3/strict-max-bean-instance-pool=demo:add(max-pool-size=10)'

JMS

Add JMS queue:

jms-queue add --queue-address= jms.queue.ExampleQueue --entries=java:/jms/queue/exampleQueue,java:/jboss/exported/jms/queue/exampleQueue

Add JMS Topic:

jms-topic add --topic-address=jms.topic.ExampleTopic --entries=java:/jms/topic/exampleTopic,java:/jboss/exported/jms/topic/exampleTopic

Transactions

How to check a transaction attribute:

/subsystem=transactions/log-store=log-store/transactions=0\:defe7f000002\:-c71efc2\:4f9e6f8f\:9:read-resource

How to view all prepared transactions:

/subsystem=transactions/log-store=log-store:read-children-names(child-type=transactions)

How to delete a transaction:

/subsystem=transactions/log-store=log-store/transactions=0\:defe7f000002\:-b66efc2\:c71efc2\:9:delete

How to recover a transaction:

/subsystem=transactions/log-store=log-store/transactions=0\:defe7f000002\:-b66efc2\:c71efc2\:9/participants=2:recover

Mixed Bash commands useful for WildFly admins

How to store in a bash variable the result of a Command Line readAttribute

heapused=$(./jboss-cli.sh -c "/core-service=platform-mbean/type=memory:read-attribute(name=heap-memory-usage)" | grep "used" | awk '{print $3}' | sed 's/L,//')

How to find HTTP requests (from access log file) that took more than 100 seconds:

egrep -R "TimeTaken: 1[0-9]{2}.[0-9]" access.log*

How to check that the application web.war is deployed

APPS=$(./jboss-cli.sh --controller=127.0.0.1:9999 -c "deploy")

list=( $APPS )

deployed=false

for i in "${list[@]}"
do
        #echo $i
        if [ $i = "web.war" ] ; then
        deployed=true;   
        fi
done

if [ "$deployed" = true ]
then
 echo "Application web.war deployed"
else
echo "Application web.war not available"
fi

JAR Files

How to find a class in a set of JARS recursively

Here is how to search recursively for the class org.jboss.as.ejb3.cache.CacheFactory in the modules folder of the application server

$ grep -r org.jboss.as.ejb3.cache.CacheFactory modules
Binary file modules/system/layers/base/org/jboss/as/ejb3/main/jboss-as-ejb3-7.5.0.Final-redhat-21.jar matches

How to compile a class adding all the application server JARS in the classpath

$ javac -cp .:`find $JBOSS_HOME -name "*.jar" | tr "\n" ":"` MyClass.java

Server logs

How to read server logs using REST

http://localhost:9990/management/

subsystem/logging/log-file/server.log?operation=attribute&name=stream&useStreamAsResponse

How to grep the Server logs for the last hour (ex. find the ERROR String)

$ grep "^$(date -d '-1 hour' +'%H')" /home/jboss/server.log  | grep 'ERROR'

How to grep the Server logs for the last minutes (ex. find the ERROR String)

$ sed -n "/^$(date --date='10 minutes ago' '+%H:%M')/,\$p" /home/jboss/log/server.log | grep "ERROR"

How to find which logs files contain a String in the last n days (ex. 5 days)

$ find . -name "server.log*" -mtime -5 | xargs grep -i "OutOfMemoryError"

Clustering

How to check that multicast is working properly

$ java -cp jgroups-3.0.14.Final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555
$ java -cp jgroups-3.0.14.Final.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555

(Replace the jgroups JAR with the one you have in your distribution)

JBoss Process

How to find JBoss PID

Option 1)

$ pgrep -f jboss
9156

Option 2)

#! /bin/bash
 if [ -z "$(ps -ef | grep org.jboss.Main)" ]
then 
 echo "jboss is not running"
else
  echo "jboss is running"
fi

Option 3) – Needs JDK !

$ jps -lv | grep -i jboss | cut -d ' ' -f 1
9156

How to perform a Thread Dump

Option 1)

$ kill -3 JBOSSPID

Option 2)

$ jstack -l JBOSSPID > jstack.out

Replace JBOSSPID with the Application server’s PID

How to count the number of Threads running

On linux architectures, you can count the number of running threads (as lightweight processes) as follows:

$ ps -eLf | grep [PID] | wc -l

How to check the CPU usage of a Thread

Thread are qualified by Linux Operating system as lightweight processes. You have to execute at first top to retrieve the CPU usage of a Thread Id:

$ top -b -n 1 -H -p JBOSSPID

Example output:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 8374 jboss      25   0 2284m 640m  80m R 39.5 23.7  13:00.64 java               
 8465 jboss      25   0 2284m 640m  80m R 19.5 23.7   7:23.10 java

Now that you have identified the top CPU consuming Thread ids (Column PID), check their stack trace to see what’s going on

How to monitor the Java Process size (using pmap)

pmap -x <pid>

How to store the amount of RSS (Process Resident Size) of a process in a variable

Assumed that the outout of pmap has been stored in a file named pmap.log:

export JVMSIZE=$(grep "total kB" pmap.log | awk '{print $4}')

How to monitor the Java Native Memory: (Requires JDK!):

Include the following settings in your JDK:

-XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics

Then, execute the following command against the Java PID:

jcmd <pid> VM.native_memory

How to monitor a Java Process Heap size: (Requires JDK!)

jmap -heap PID

Example:

$ jmap -heap 713
Attaching to process ID 713, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.60-b23

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 1367343104 (1304.0MB)
   NewSize                  = 22020096 (21.0MB)
   MaxNewSize               = 455606272 (434.5MB)
   OldSize                  = 45088768 (43.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 84934656 (81.0MB)
   used     = 40320120 (38.45226287841797MB)
   free     = 44614536 (42.54773712158203MB)
   47.471929479528356% used
From Space:
   capacity = 18874368 (18.0MB)
   used     = 0 (0.0MB)
   free     = 18874368 (18.0MB)
   0.0% used
To Space:
   capacity = 23592960 (22.5MB)
   used     = 0 (0.0MB)
   free     = 23592960 (22.5MB)
   0.0% used
PS Old Generation
   capacity = 91226112 (87.0MB)
   used     = 36171208 (34.49555206298828MB)
   free     = 55054904 (52.50444793701172MB)
   39.65005984251527% used

Networking

How to monitor incoming traffic using tcpdump

Example: monitor port 9990

$ tcpdump -A -i lo 'port 9990' -v

Example: monitor port 9990 on host 127.0.0.1:

$ tcpdump -A -i lo 'dst host 127.0.0.1 and dst port 9990' -v

Example: monitor multicast on the current machine:

$ tcpdump -n “multicast”

How to tunnel some ports of the application server with SSH

Example: tunnel port 9990 of the IP Address 192.168.10.1 on localhost (so for example you can reach the remote console with http://localhost:9990)

$ ssh -R 9990:192.168.10.1:9990

userAT192.168.10.1

How to check ports opened by a JBoss/WildFly process

$ lsof -Pri | grep java
java      9156 francesco  343u  IPv4  92363      0t0  TCP localhost.localdomain:8080 (LISTEN)
java      9156 francesco  380u  IPv4  92366      0t0  TCP localhost.localdomain:9990 (LISTEN)

How to find which process is engaging my JBoss ports: (e.g. 8080)

$ netstat -anp | grep 8080
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      3861/ssh

Please note that you need a super-user if you want to see all processes running on the machine. In our case the PID is 3861 used by ssh

How to generate a virtual IP Address (Useful for clustering purposes)

$ ifconfig eth0:2 10.0.1.1 netmask 255.255.255.0

IO and Disk Space

How to find the size of a directory, including subdirectories (Especially useful for log files)

$ du -sh standalone/log/
600K	standalone/log/

How to find space left on a device

$ df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   50G   27G   20G  58% /

How to compare two files side by side (useful to check differences between configuration files)

$ sdiff -s standalone1.xml standalone2.xml
<connection-url>jdbc:h2:mem:test</connection-url> | <connection-url>jdbc:h2:mem:other</connection-url

How to find IO statistics per process (useful especially for journal!)

$ cat /proc/4383/io 
rchar: 6311275
wchar: 6525
syscr: 3883
syscw: 40
read_bytes: 3129344
write_bytes: 77688832
cancelled_write_bytes: 0

In the above example, we are checking IO statistics of the process id 4383.

Backup and Recovery

How to backup a JBoss installation with rsync

rsync -azR --delete --exclude "data/activemq" --exclude "data/kernel" --exclude "data/timer-service-data" --exclude "data/tx-object-store" --exclude "log/*" --exclude "tmp/*" /home/jboss/jboss-eap-7.0 ServerDest:/path

In this example, we are syncing the folder /home/jboss/jboss-eap-7.0 to the Host “ServerDest” and path: “path“. Exclude log and tmp files and some data files.

Jakarta EE

How to recursively replace all javax occurrences with jakarta in Java classes.

find . -name '*.java' | xargs sed -i 's/javax/jakarta/g'

Web services

How to call a REST service with curl using authentication and passing JSON parameters

$ curl -u admin:admin -H "Content-Type: application/json" -d '{"name":"John Doe", "email": "john.doeATjboss.com", "phoneNumber":"11223344"}' http://localhost:8080/restservice