The java.lang.OutOfMemoryError: Metaspace indicates that the amount of native memory allocated for Java class metadata is exausted. Let’s how this issue can be solved in standalone applications and cloud applications.
In general terms, the more classes you are loading into the JVM, the more memory will be consumed by the metaspace. In Java 8 and later, the maximum amount of memory allocated for Java classes (MaxMetaspaceSize) is by default unlimited, so in most cases there is no need to change this setting.
On the other hand, if you want to limit the amount of memory allocated for Java classes, you can set it as follows:
java -XX:MaxMetaspaceSize=3200m
The current Metaspace size (i.e. committed) will be, generally, smaller than that. Why ? Because there is a setting, MaxMetaspaceFreeRatio, which prevents the Metaspace from growing too much.
MaxMetaspaceFreeRatio is a JVM option that sets the percentage of free space that the JVM should attempt to maintain in the metaspace. When the percentage of free space in the metaspace drops below this threshold, the JVM will attempt to free up space by unloading classes that are no longer in use. The default value of MaxMetaspaceFreeRatio is 70.
In conclusion, setting a high value for MaxMetaspaceSize, increase the chances of OutOfMemoryError: Metaspace
What is the solution ?
By increasing the upper limit of the attribute MaxMetaspaceSize might solve the issue in some cases. For example, if you can see the error upon application start up.
On the other hand, if you see this error at runtime, we recommend to investigate on the Root cause of the problem before increasing blindly the MaxMetaspaceSize.
In order to find the Root cause, we recommend that you follow these steps:
- Understand how the JVM manages the MetaSpace data
- Learn how to monitor the MetaSpace in a running Java application
- Check the Heap dump of your application to find offending Classes
How Java Hotspot manages MetaSpace Data
The Java Hotspot manages Metadata space as follows: firstly, the JVM requests an amount of space to the OS. Then, this space is divided into chunks. A class loader allocates space for Metadata from its chunks.
Class Metadata is deallocated when the corresponding Java class is unloaded and its chunks are recycled for reuse or returned to the OS. Java classes are unloaded as a result of Garbage collection, and garbage collections may be triggered in order to unload classes and deallocate class Metadata. When the space committed for class Metadata reaches a certain threshold (a high-water mark), a Garbage Collection happens.
After the Garbage Collection, the high-water mark may change depending on the amount of space freed from class Metadata.
Monitoring the MetaSpace with jstat
The simplest way to monitor the MetaSpace size is by means of the jstat tool which is available in the JDK. When used with the option -gcmetacapacity it provides the following information:
jstat -gcmetacapacity (PID)
For example:
MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC FGCT CGC CGCT 0.0 374784.0 140360.0 0.0 253952.0 21168.0 23 0 0.000 6 0.046
And here is a description of the Labels:
- MCMN: Minimum metaspace capacity (kB).
- MCMX: Maximum metaspace capacity (kB).
- MC: Metaspace capacity (kB).
- CCSMN: Compressed class space minimum capacity (kB).
- CCSMX: Compressed class space maximum capacity (kB).
- YGC: Number of young generation GC events.
- FGC: Number of full GC events.
- FGCT: Full garbage collection time.
- GCT: Total garbage collection time.
Besides the Metaspace capacity, pay attention to the Compressed class space maximum capacity. The CompressedClassSpaceSize is the amount of virtual memory reserved for compressing Class MetaData. Therefore, MaxMetaspaceSize should be larger than CompressedClassSpaceSize.
If MaxMetaspaceSize is set smaller than CompressedClassSpaceSize, the JVM auto adjusts CompressedClassSpaceSize using this formula:
CompressedClassSpaceSize = MaxMetaspaceSize - 2 * InitialBootClassLoaderMetaspaceSize
Other interesting options include the parameter -gcutil:
$ jstat -gcutil (PID) | awk '{print($5)}'
This will print the Metaspace utilization as a percentage of the space’s current capacity.
Monitoring the MetaSpace with Java Native Memory tracking
Another option to monitor the MetaSpace usage is the JVM parameter NativeMemoryTracking, which you can add to the start up parameters. For example:
-XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics
When you enable the Native Memory Tracking, you can request a report on the JVM memory usage using the following command:
$ jcmd <pid> VM.native_memory
If you check at the jcmd output, you will find at the bottom, the amount of native memory committed in the Internal (committed) section
Total: reserved=1334532KB, committed=369276KB - Java Heap (reserved=524288KB, committed=132096KB) (mmap: reserved=524288KB, committed=132096KB) - Class (reserved=351761KB, committed=112629KB) (classes #19111) ( instance classes #17977, array classes #1134) (malloc=3601KB #66765) (mmap: reserved=348160KB, committed=109028KB) ( Metadata: ) ( reserved=94208KB, committed=92824KB) ( used=85533KB) ( free=7291KB) ( waste=0KB =0.00%) ( Class space:) ( reserved=253952KB, committed=16204KB) ( used=12643KB) ( free=3561KB) ( waste=0KB =0.00%)
In the line beginning with Metaspace, look for the used value. This is the amount of space the JVM uses for loaded classes. The committed value is the amount of space available for chunks. The reserved value is the amount of space reserved (but not necessarily committed) for metadata.
Querying the Meta Space from an Heap Dump
To have more insights on the MetaSpace error, you should inspect the Heap of your Java application. You can use the jmap command line to trigger a Heap Dump:
jmap -dump:format=b,file=dump.hprof <PID>
Then, open the Heap Dump with your favourite tool. For example Eclipse MAT:
From the Heap Dump, you can look for duplicate classes, especially those loading your application classes.
The simplest way to do that, is using the OQL console. Within the OQL Console you can execute OQL queries to perform ad hoc analysis on your classes. For example, by executing the following query, you can have a list of class loaded from each Classloader:
select map(sort(map(heap.objects('java.lang.ClassLoader'), '{loader: it, count: it.classes.elementCount }'), 'lhs.count < rhs.count'), 'toHtml(it) + "
"')
This can be a precious hint to determine if a Classloader is loading an increasing number of classes.
OutOfMemoryError: Metaspace on OpenShift/Kubernetes
When using OpenJDK Image on OpenShift/Kubernetes, the default maximum value for the Metaspace is XX:MaxMetaspaceSize=100m. You might have noticed that setting this value through the JAVA_OPTIONS environment variable, doesn’t work as the default value is appended to the bottom:
VM Arguments: -Xms128m -Xmx1024m -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256m -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError
The correct way to set the MaxMetaspaceSize is through the GC_MAX_METASPACE_SIZE environment variable. For example, if you are using a deployment.yaml file to deploy your application with JKube, the following settings will override the default values for the MaxMetaspaceSize and MaxMetaspaceSize:
spec:
template:
spec:
containers:
- env:
- name: JAVA_OPTIONS
value: '-Xms128m -Xmx1024m'
- name: GC_MAX_METASPACE_SIZE
value: 256
- name: GC_METASPACE_SIZE
value: 96
