JBoss cheatsheet for Linux Administrators

This is my JBoss / WildFly cheatsheet I use for managing the application server on a Linux box. Enjoy it and welcome any contribution to the cheatsheet.

Domain

How to check that the Domain Controller is alive

#!/bin/bash
CONNECTED=$(netstat -an | grep -wF 9999 | wc -l)
while [ $CONNECTED -gt 0 ]
do
  sleep 20
  CONNECTED=$(netstat -an | grep -wF 9999 | wc -l)
done
echo "Lost Domain Controller! Do something here."

Replace "9999" with the jboss.management.native.port if you don't use the default port.

JBoss CLI

How to store in a bash variable the result of a Command Line readAttribute

heapused=$(./jboss-cli.sh -c "/core-service=platform-mbean/type=memory:read-attribute(name=heap-memory-usage)" | grep "used" | awk '{print $3}' | sed 's/L,//')

How to check that the application web.war is deployed

APPS=$(./jboss-cli.sh --controller=127.0.0.1:9999 -c "deploy")

list=( $APPS )

deployed=false

for i in "${list[@]}"
do
        #echo $i
        if [ $i = "web.war" ] ; then
        deployed=true;   
        fi
done

if [ "$deployed" = true ]
then
 echo "Application web.war deployed"
else
echo "Application web.war not available"
fi

How to shutdown the application server

$JBOSS_HOME/bin/jboss-cli.sh --connect --controller=$CONTROLLER_IP:$CONTROLLER_PORT --command=:shutdown

JAR Files

How to find a class in a set of JARS recursively

Here is how to search recursively for the class org.jboss.as.ejb3.cache.CacheFactory in the modules folder of the application server

$ grep -r org.jboss.as.ejb3.cache.CacheFactory modules
Binary file modules/system/layers/base/org/jboss/as/ejb3/main/jboss-as-ejb3-7.5.0.Final-redhat-21.jar matches

How to compile a class adding all the application server JARS in the classpath

$ javac -cp .:`find $JBOSS_HOME -name "*.jar" | tr "\n" ":"` MyClass.java

Server logs

How to grep the Server logs for the last hour (ex. find the ERROR String)

$ grep "^$(date -d '-1 hour' +'%H')" /home/jboss/server.log  | grep 'ERROR'

How to grep the Server logs for the last minutes (ex. find the ERROR String)

$ sed -n "/^$(date --date='10 minutes ago' '+%H:%M')/,\$p" /home/jboss/log/server.log | grep "ERROR"

How to find which logs files contain a String in the last n days (ex. 5 days)

$ find . -name "server.log*" -mtime -5 | xargs grep -i "OutOfMemoryError"

Clustering

How to check that multicast is working properly

$ java -cp jgroups-3.0.14.Final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555
$ java -cp jgroups-3.0.14.Final.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555

(Replace the jgroups JAR with the one you have in your distribution)

JBoss Process

How to find JBoss PID

Option 1)

$ pgrep -f jboss
9156

Option 2)

#! /bin/bash
 if [ -z "$(ps -ef | grep org.jboss.Main)" ]
then 
 echo "jboss is not running"
else
  echo "jboss is running"
fi

Option 3) - Needs JDK !

$ jps -lv | grep -i jboss | cut -d ' ' -f 1
9156

How to perform a Thread Dump

Option 1)

$ kill -3 JBOSSPID

Option 2)

$ jstack -l JBOSSPID > jstack.out

Replace JBOSSPID with the Application server's PID

How to count the number of Threads running

On linux architectures, you can count the number of running threads (as lightweight processes) as follows:

$ ps -eLf | grep [PID] | wc -l

How to check the CPU usage of a Thread

Thread are qualified by Linux Operating system as lightweight processes. You have to execute at first top to retrieve the CPU usage of a Thread Id:

$ top -b -n 1 -H -p JBOSSPID

Example output:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 8374 jboss      25   0 2284m 640m  80m R 39.5 23.7  13:00.64 java               
 8465 jboss      25   0 2284m 640m  80m R 19.5 23.7   7:23.10 java  

Now that you have identified the top CPU consuming Thread ids (Column PID), check their stack trace to see what's going on

How to monitor a Java Process Heap size: (Requires JDK!)

jmap -heap PID

Example:

$ jmap -heap 713
Attaching to process ID 713, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.60-b23

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 1367343104 (1304.0MB)
   NewSize                  = 22020096 (21.0MB)
   MaxNewSize               = 455606272 (434.5MB)
   OldSize                  = 45088768 (43.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 84934656 (81.0MB)
   used     = 40320120 (38.45226287841797MB)
   free     = 44614536 (42.54773712158203MB)
   47.471929479528356% used
From Space:
   capacity = 18874368 (18.0MB)
   used     = 0 (0.0MB)
   free     = 18874368 (18.0MB)
   0.0% used
To Space:
   capacity = 23592960 (22.5MB)
   used     = 0 (0.0MB)
   free     = 23592960 (22.5MB)
   0.0% used
PS Old Generation
   capacity = 91226112 (87.0MB)
   used     = 36171208 (34.49555206298828MB)
   free     = 55054904 (52.50444793701172MB)
   39.65005984251527% used

Networking

How to monitor incoming traffic using tcpdump

Example: monitor port 9990

$ tcpdump -A -i lo 'port 9990' -v

Example: monitor port 9990 on host 127.0.0.1:

$ tcpdump -A -i lo 'dst host 127.0.0.1 and dst port 9990' -v

Example: monitor multicast on the current machine:

$ tcpdump -n “multicast”

How to tunnel some ports of the application server with SSH

Example: tunnel port 9990 of the IP Address 192.168.10.1 on localhost (so for example you can reach the remote console with http://localhost:9990)

$ ssh -R 9990:192.168.10.1:9990 userAT192.168.10.1

How to check ports opened by a JBoss/WildFly process

$ lsof -Pri | grep java
java      9156 francesco  343u  IPv4  92363      0t0  TCP localhost.localdomain:8080 (LISTEN)
java      9156 francesco  380u  IPv4  92366      0t0  TCP localhost.localdomain:9990 (LISTEN)

How to find which process is engaging my JBoss ports: (e.g. 8080)

$ netstat -anp | grep 8080
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      3861/ssh  

Please note that you need a super-user if you want to see all processes running on the machine. In our case the PID is 3861 used by ssh

How to generate a virtual IP Address (Useful for clustering purposes)

$ ifconfig eth0:2 10.0.1.1 netmask 255.255.255.0

IO and Disk Space

How to find the size of a directory, including subdirectories (Especially useful for log files)

$ du -sh standalone/log/
600K	standalone/log/

How to find space left on a device

$ df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   50G   27G   20G  58% /

How to compare two files side by side (useful to check differences between configuration files)

$ sdiff -s standalone1.xml standalone2.xml
<connection-url>jdbc:h2:mem:test</connection-url> | <connection-url>jdbc:h2:mem:other</connection-url

How to find IO statistics per process (useful especially for journal!)

$ cat /proc/4383/io 
rchar: 6311275
wchar: 6525
syscr: 3883
syscw: 40
read_bytes: 3129344
write_bytes: 77688832
cancelled_write_bytes: 0

In the above example, we are checking IO statistics of the process id 4383.

Backup and Recovery

How to backup a JBoss installation with rsync

rsync -azR --delete --exclude "data/activemq" --exclude "data/kernel" --exclude "data/timer-service-data" --exclude "data/tx-object-store" --exclude "log/*" --exclude "tmp/*" /home/jboss/jboss-eap-7.0 ServerDest:/path

In this example, we are syncing the folder /home/jboss/jboss-eap-7.0 to the Host "ServerDest" and path: "path". Exclude log and tmp files and some data files.

Web services

How to call a REST service with curl using authentication and passing JSON parameters

$ curl -u admin:admin -H "Content-Type: application/json" -d '{"name":"John Doe", "email": "john.doeATjboss.com", "phoneNumber":"11223344"}' http://localhost:8080/restservice

Vim tricks for Admins

How to perform case insensitive searches in file

:set noignorecase 

How to split editing window and open integrated File Explorer

:Sex

How to keep file updated (e.g.log file)

:set autoread

How to show line numbers in a file

:set number

How to replace all String old with new:

:%s/old/new/g

How to format an XML file:

First include in your ~/.vimrc:

filetype indent on
set smartindent

Then, open your file with vim and type:

gg=G

Related articles available on mastertheboss.com

How do I get the list of MBeans with twiddle?

  JBoss provides a simple command line tool that allows for inte

How to count active session with twiddle ?

  There are some well known alternatives to count the number of

How do I run garbage collector from shell?

  You can suggest JBoss to run the garbage collector in differen

How do I check my application status from shell?

  This is a two process step: at first you need to retrieve the

How do you shut down multiple JBoss instances?

  Supposing you have configured 2 instances of JBoss, one on por

twiddle reference guide

Please note: twiddle is not available any more as default Command

Follow us on Twitter