Solving java.lang.OutOfMemoryError: Metaspace error

The java.lang.OutOfMemoryError: Metaspace indicates that the amount of native memory allocated for Java class metadata is exausted. Let’s how this issue can be solved in standalone applications and cloud applications.

In Java 8 and later, the maximum amount of memory allocated for Java classes (MaxMetaspaceSize) is by default unlimited, so in most cases there is no need to change this setting. On the other hand, if you want to limit the amount of memory allocated for Java classes, you can set it as follows:

java -XX:MaxMetaspaceSize=3200m  

The thing is that -XX:MaxMetaspaceSize is just an upper limit. The current Metaspace size (i.e. committed) will be smaller. In fact, there is a setting called MaxMetaspaceFreeRatio (default 70%) which means that the actual metaspace size will never exceed 230% of its occupancy.

And for it to grow it first would have to fill up, forcing a garbage collection in an attempt to free objects and only when it cannot meet its MinMetaspaceFreeRatio (default 40%) goal it would expand the current metaspace. That can however not be greater than 230% of the occupancy after the GC cycle.

How Java Hotspot manages MetaSpace Data

The Java Hotspot manages the space used for metadata as follows: space is requested from the OS and then divided into chunks. A class loader allocates space for metadata from its chunks.

java outofmemory metaspace

Class metadata is deallocated when the corresponding Java class is unloaded and its chunks are recycled for reuse or returned to the OS. Java classes are unloaded as a result of garbage collection, and garbage collections may be triggered in order to unload classes and deallocate class metadata. When the space committed for class metadata reaches a certain threshold (a high-water mark), a garbage collection is triggered.
After the garbage collection, the high-water mark may be raised or lowered depending on the amount of space freed from class metadata.

Checking MetaSpace capacity with jstat

The simplest way to monitor the MetaSpace size is by means of the jstat tool which is available in the JDK. When used with the option -gcmetacapacity it provides the following information:

jstat -gcmetacapacity (PID)  

For example:

MCMN   MCMX      MC       CCSMN CCSMX       CCSC    YGC   FGC    FGCT    CGC    CGCT       
0.0   374784.0  140360.0  0.0   253952.0    21168.0  23     0    0.000     6    0.046   

And here is a description of the Labels:

  • MCMN: Minimum metaspace capacity (kB).
  • MCMX: Maximum metaspace capacity (kB).
  • MC: Metaspace capacity (kB).
  • CCSMN: Compressed class space minimum capacity (kB).
  • CCSMX: Compressed class space maximum capacity (kB).
  • YGC: Number of young generation GC events.
  • FGC: Number of full GC events.
  • FGCT: Full garbage collection time.
  • GCT: Total garbage collection time.

Other interesting options include the parameter -gcutil:

 $ jstat -gcutil (PID) | awk '{print($5)}' 

This will print the Metaspace utilization as a percentage of the space’s current capacity.

Querying the Meta Space from an Heap Dump

Further inspection can be performed through an Heap Dump:

solving java.lang.OutOfMemoryError: Metaspace error java.lang.OutOfMemoryError: Metaspace

Then, if you have a look at the OQL console, you can execute OQL queries to perform ad hoc analysis on your classes. For example, by executing the following query, you can have a list of class loaded from each Classloader:

select map(sort(map(heap.objects('java.lang.ClassLoader'), '{loader: it, count: it.classes.elementCount }'), 'lhs.count < rhs.count'), 'toHtml(it) + "
"')

solving java.lang.OutOfMemoryError: Metaspace error java.lang.OutOfMemoryError: Metaspace

This can be a precious hint to determine if a Classloader is loading an increasing number of classes.

Monitoring MetaSpace Size with Java Native Memory tracking

A good way to monitor the exact amount of Metadata is by using the NativeMemoryTracking, which can be added through the following settings:

-XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics

When Native Memory Tracking is enabled, you can request a report on the JVM memory usage using the following command:

$ jcmd <pid> VM.native_memory

If you check at the jcmd output, you will find at the bottom, the amount of native memory committed/used in the Internal (committed) section

Total: reserved=1334532KB, committed=369276KB
-                 Java Heap (reserved=524288KB, committed=132096KB)
                            (mmap: reserved=524288KB, committed=132096KB)
 
-                     Class (reserved=351761KB, committed=112629KB)
                            (classes #19111)
                            (  instance classes #17977, array classes #1134)
                            (malloc=3601KB #66765)
                            (mmap: reserved=348160KB, committed=109028KB)
                            (  Metadata:   )
                            (    reserved=94208KB, committed=92824KB)
                            (    used=85533KB)
                            (    free=7291KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=253952KB, committed=16204KB)
                            (    used=12643KB)
                            (    free=3561KB)
                            (    waste=0KB =0.00%)

In the line beginning with Metaspace, the used value is the amount of space used for loaded classes. The committed value is the amount of space available for chunks. The reserved value is the amount of space reserved (but not necessarily committed) for metadata.

OutOfMemoryError: Metaspace on OpenShift/Kubernetes

When using openjdk Image on OpenShift/Kubernetes, the default maxium value for the Metaspace is XX:MaxMetaspaceSize=100m. You might have noticed that setting this value through the JAVA_OPTIONS environment variable, doesn’t work as the default value is appended to the bottom:

VM Arguments: -Xms128m -Xmx1024m -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256m    -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError

The correct way to set the MaxMetaspaceSize is through the GC_MAX_METASPACE_SIZE environment variable. For example, if you are using a deployment.yaml file to deploy your application with JKube, the following settings will override the default values for the MaxMetaspaceSize and MaxMetaspaceSize:

spec:
  template:
    spec:
      containers:
      - env:
        - name: JAVA_OPTIONS
          value: '-Xms128m -Xmx1024m'
        - name: GC_MAX_METASPACE_SIZE
          value: 256
        - name: GC_METASPACE_SIZE
          value: 96