Solving java.lang.OutOfMemoryError: Metaspace error

The java.lang.OutOfMemoryError: Metaspace indicates that the JVM Metaspace area in memory is exhausted. Metaspace is where the JVM stores Class metadata and related information. In this article we will learn how to solve and prevent this issue.

Overview of the JVM Metaspace

To understand the java.lang.OutOfMemoryError: Metaspace let’s see where this area is located within the JVM:

java.lang.OutOfMemoryError: Metaspace solved

Since classes are loaded dynamically by the JVM during program execution, metadata for each class, method, and other runtime artifacts are generated and stored in the Metaspace. Unlike the older PermGen, Metaspace dynamically adjusts its size based on the application’s demands and is garbage-collected by the JVM when classes are unloaded or metadata becomes obsolete

The JVM options that have impact on the Metaspace are the following ones:

java -XX:MaxMetaspaceSize=256M -XX:MetaspaceSize=128M -XX:MaxMetaspaceFreeRatio=60 YourMainClass
  • -XX:MaxMetaspaceSize=256M: Sets the maximum size of the Metaspace to 256 megabytes.
  • -XX:MetaspaceSize=128M: Sets the initial size of the Metaspace to 128 megabytes.
  • -XX:MaxMetaspaceFreeRatio=60: Sets the maximum Metaspace free ratio to 60%. When the percentage of free space in the metaspace drops below this threshold, the JVM will attempt to free up space by unloading classes that are no longer in use.

Solving java.lang.OutOfMemoryError: Metaspace

The first and obvious recommendation is to remove the -XX:MaxMetaspaceSize option and let the JVM manage the metaspace size automatically.

As you can check from your JVM options, in modern JVMs the size of the MaxMetaspaceSize is unlimited:

java -XX:+PrintFlagsFinal | grep MaxMetaspace
   size_t MaxMetaspaceExpansion                    = 5439488                                    
    uintx MaxMetaspaceFreeRatio                    = 70                                       
   size_t MaxMetaspaceSize                         = 18446744073709551615   

The simplest scenario is that you are setting a -XX:MaxMetaspaceSize which is too low for your application requirements.

On the other hand, if you are not setting a low value for the you should inspect your JVM to find the Root cause. For example, the following graph shows that upon each application deployment, the Metaspace slightly increases. In the long run this can lead to the exaustion of the Metaspace

How to fix java.lang.OutOfMemoryError: Metaspace

An increase in the Metaspace should be in direct relation with the number of Classes loaded by the JVM. Here is a clear example of an issue with the loading of Classes

java.lang.OutOfMemoryError: Metaspace solve

To find the root cause, we need to follow the following steps:

  • Learn how to monitor the Metaspace to see if the amount grows indefinitely
  • Then, understand which are the classes that your JVM is not able to release

Before getting into the resolution, for testing purpose you can find a sample code to reproduce the java.lang.OutOfMemoryError: Metaspace in our repository: Metaspace OOM reproducer

Step 1: Monitoring the MetaSpace size

The simplest way to monitor the MetaSpace size is by means of the jstat tool which is available in the JDK. When used with the option -gcmetacapacity it provides the following information:

jstat -gcmetacapacity (PID)  

For example:

MCMN   MCMX      MC       CCSMN CCSMX       CCSC    YGC   FGC    FGCT    CGC    CGCT       
0.0   374784.0  140360.0  0.0   253952.0    21168.0  23     0    0.000     6    0.046   

And here is a description of the Labels:

  • MCMN: Minimum metaspace capacity (kB).
  • MCMX: Maximum metaspace capacity (kB).
  • MC: Metaspace capacity (kB).
  • CCSMN: Compressed class space minimum capacity (kB).
  • CCSMX: Compressed class space maximum capacity (kB).
  • YGC: Number of young generation GC events.
  • FGC: Number of full GC events.
  • FGCT: Full garbage collection time.
  • GCT: Total garbage collection time.

Besides the Metaspace capacity, pay attention to the Compressed class space maximum capacity. The CompressedClassSpaceSize is the amount of virtual memory reserved for compressing Class MetaData. Therefore, MaxMetaspaceSize should be larger than CompressedClassSpaceSize.

If MaxMetaspaceSize is set smaller than CompressedClassSpaceSize, the JVM auto adjusts CompressedClassSpaceSize using this formula:

CompressedClassSpaceSize = MaxMetaspaceSize - 2 * InitialBootClassLoaderMetaspaceSize

For instance, consider the following settings, resulting in a 768M space for class metadata, with 756M (assuming an initial boot class loader Metaspace size of 6M) allocated for compressed class pointers:

-XX:MetaspaceSize=768M -XX:MaxMetaspaceSize=768M -XX:CompressedClassSpaceSize=1536M

Other interesting options include the parameter -gcutil:

 $ jstat -gcutil (PID) | awk '{print($5)}' 

This will print the Metaspace utilization as a percentage of the space’s current capacity.

Monitoring the MetaSpace with Java Native Memory tracking

Another option to monitor the MetaSpace usage is the JVM parameter NativeMemoryTracking, which you can add to the start up parameters. For example:

-XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics

When you enable the Native Memory Tracking, you can request a report on the JVM memory usage using the following command:

$ jcmd <pid> VM.native_memory

If you check at the jcmd output, you will find at the bottom, the amount of native memory committed in the Internal (committed) section

Total: reserved=1334532KB, committed=369276KB
-                 Java Heap (reserved=524288KB, committed=132096KB)
                            (mmap: reserved=524288KB, committed=132096KB)
 
-                     Class (reserved=351761KB, committed=112629KB)
                            (classes #19111)
                            (  instance classes #17977, array classes #1134)
                            (malloc=3601KB #66765)
                            (mmap: reserved=348160KB, committed=109028KB)
                            (  Metadata:   )
                            (    reserved=94208KB, committed=92824KB)
                            (    used=85533KB)
                            (    free=7291KB)
                            (    waste=0KB =0.00%)
                            (  Class space:)
                            (    reserved=253952KB, committed=16204KB)
                            (    used=12643KB)
                            (    free=3561KB)
                            (    waste=0KB =0.00%)

In the line beginning with Metaspace, look for the used value. This is the amount of space the JVM uses for loaded classes. The committed value is the amount of space available for chunks. The reserved value is the amount of space reserved (but not necessarily committed) for metadata.

Step 2: Find the Top loaded classes

Ok, at this point you know that your Metaspace is growing too much. The next step is to find which Classes are responsible for the abnormal growth of the Metaspace. In order to do that, we need an Heap of your Java application.

If you want to know more about generating an Heap Dump, check this article: How to generate an Heap Dump in Java

In a nutshell, can use the jmap command line to trigger a Heap Dump:

jmap -dump:format=b,file=dump.hprof <PID>

Then, open the Heap Dump with your favourite tool. For example Eclipse MAT:

java.lang.OutOfMemoryError: Metaspace error

From the Heap Dump, you can look for duplicate classes, especially those loading your application classes.

The simplest way to do that, is using the OQL console. Within the OQL Console you can execute OQL queries to perform ad hoc analysis on your classes. For example, by executing the following query, you can have a list of class loaded from each Classloader:

select map(sort(map(heap.objects('java.lang.ClassLoader'), '{loader: it, count: it.classes.elementCount }'), 'lhs.count < rhs.count'), 'toHtml(it) + "
"')
solving java.lang.OutOfMemoryError: Metaspace error

Additionally, to have more insights on WildFly, you can run this OQL which is specific to jboss Classes:

SELECT module.name.value.toString() FROM org.jboss.modules.ModuleClassLoader 

Finally, if you want to use just the command line, another option is to add the -XX:+UnlockDiagnosticVMOptions at startup.

Then, execute the jcmd command passing as argument the GC.class_stats:

jcmd <PID> GC.class_stats -all -csv > class_stats.csv

The above command, will produce a CSV file. Check the InstBytes column to see which Classes are retaining more memory.

OutOfMemoryError: Metaspace on OpenShift/Kubernetes

When using OpenJDK Image on OpenShift/Kubernetes, the default maximum value for the Metaspace is XX:MaxMetaspaceSize=100m. You might have noticed that setting this value through the JAVA_OPTIONS environment variable, doesn’t work as the default value is appended to the bottom:

VM Arguments: -Xms128m -Xmx1024m -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256m    -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError

The correct way to set the MaxMetaspaceSize is through the GC_MAX_METASPACE_SIZE environment variable. For example, if you are using a deployment.yaml file to deploy your application with JKube, the following settings will override the default values for the MaxMetaspaceSize and MaxMetaspaceSize:

spec:
  template:
    spec:
      containers:
      - env:
        - name: JAVA_OPTIONS
          value: '-Xms128m -Xmx1024m'
        - name: GC_MAX_METASPACE_SIZE
          value: 256
        - name: GC_METASPACE_SIZE
          value: 96

Conclusion

In conclusion, resolving the java.lang.OutOfMemoryError related to Metaspace requires a multifaceted approach that involves understanding, monitoring, and optimizing Metaspace usage in Java applications.

Found the article helpful? if so please follow us on Socials