Comments (4)
@dmitripivkine pls take a look
from openj9.
Adding this item to GC queue.
OOM is triggered by Excessive GC condition. However there are 673 free regions (of 2022 total), 1m each.
This does not look right, we need to look closer.
from openj9.
There is the problem with Balanced GC - calculation of current Eden size converged to minimum value of two regions and got stuck there. This causes PGCs triggered by Allocation Failure almost back to back up to 95% time GC utilization and Excessive GC condition latch eventually. We are investigating.
from openj9.
Some initial observations.
We seem to be constantly doing PGCs, and that's because Eden is only 2MBs (2 regions of 1MB). We should be expanding the Eden since the heap is 2GB and fully expanded. But for some reason we don't expand.
The core logic for Eden expansion (for the case of fully expanded heap) is in calculateRecommendedEdenChangeForExpandedHeap, which has some non-trivai calculatinos. Unfortuantelly a couple of trace point in that method did not show up in the dump (even though one of them is Level 1!?)
However, there are quite a few relevant parameters in SchedulingDelegate, and most of them look reasonable, except maybe _estimatedFreeTenure of about 8MB, which seems rather low (trace points report around 700MB)
> !mm_schedulingdelegate 0x00007FFF90069338 | grep _estimatedFreeTenure
0x100: U64 _estimatedFreeTenure = 0x000000000083F133 (8646963)
20:33:12.362818623 0x0 j9mm.346 Event PGC end: workstackoverflow=0 overflowcount=0 heapFree=715128832/2120220672
20:33:12.365062888 0x0 j9mm.345 Event PGC start: globalcount=4335 nextGMPIncrement=0
20:33:12.442877284 0x0 j9mm.346 Event PGC end: workstackoverflow=0 overflowcount=0 heapFree=714080256/2120220672
20:33:12.444926051 0x0 j9mm.345 Event PGC start: globalcount=4336 nextGMPIncrement=0
GC does observe high PGC over head (more like 84-89%, but still very high and we should expand Eden)
> !mm_schedulingdelegate 0x00007FFF90069338 | grep _historicalPartialGCTime
0x128: U64 _historicalPartialGCTime = 0x0000000000000043 (67)
> !mm_schedulingdelegate 0x00007FFF90069338 | grep _averagePgcInterval
0x180: U64 _averagePgcInterval = 0x00000000000129ED (76269)
(lldb) p 67 * 1000.0 / 76269
(double) 0.87846962724042532
> !mm_schedulingdelegate 0x00007FFF90069338 | grep _partialGcOverhead
0x120: double _partialGcOverhead = 0.8466813906537427
Evidence that Eden is stuck at 2 regions:
> !mm_schedulingdelegate 0x00007FFF90069338 | grep -i edenregion
0x30: U64 _idealEdenRegionCount = 0x0000000000000002 (2)
0x38: U64 _minimumEdenRegionCount = 0x0000000000000001 (1)
0x40: U64 _edenRegionCount = 0x0000000000000002 (2)
0x108: U64 _maxEdenRegionCount = 0x00000000000005EC (1516)
0x110: U64 _minEdenRegionCount = 0x0000000000000002 (2)
Evidence that heap is fully expanded:
> !MM_GCExtensions 0x00007FFF90059500 |grep memoryMax
0x5fd8: U64 memoryMax = 0x000000007E600000 (2120220672)
> !mm_schedulingdelegate 0x00007FFF90069338 | grep -i _numberOfHeapRegions
0x58: U64 _numberOfHeapRegions = 0x00000000000007E6 (2022)
(lldb) p 2022 * 1024 * 1024
(int) 2120220672
from openj9.
Related Issues (20)
- cmdLineTester_criu_nonPortableRestore NotCheckpointSafeDeadlock failed HOT 3
- JDK21 compiler debug build fails due to assertion omr/compiler/x/codegen/OMRRegisterDependency.hpp:208: newCursor <= _numPostConditions HOT 1
- The KeepJNIIDs JVM arg seems to be ineffective for JDK 21+ HOT 3
- JVM Crash: hashClassTableAt: Unaligned class value - ASSERTION FAILED openj9\runtime\vm\KeyHashTable.c:73 HOT 3
- jdk_security3_0_FAILED javax/net/ssl/TLS/TLSDataExchangeTest.java ClassCastException: sun.security.rsa.RSAPublicKeyImpl incompatible with java.security.interfaces.RSAPrivateKey
- DaaLoadTest_all_special_5m_0_FAILED testNegativeInfinity(net.openj9.test.binaryData.TestFloat2ByteArray): NullPointerException
- HCR Redefinition of empty constructors gets ignored HOT 1
- DaaLoadTest_all_special_5m_9_FAILED testLessThanOrEqualsPackedDecimal(net.openj9.test.arithmetics.TestArithmeticInline):ClassCastException HOT 1
- DaaLoadTest_daa1_CS_5m_0_FAILED Segmentation error vmState=0x00000000 Compiled_method=net/openj9/test/arithmetics/TestPerformance.generateValues(II)V
- serviceability_jvmti_j9_1_FAILED serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java Error in JVMTI SetEventNotificationMode: disable SINGLE_STEP
- openJcePlusTests_0_FAILED ibm.jceplus.junit.openjceplus.multithread.TestAES_128 detail "java/lang/OutOfMemoryError", exception "native memory exhausted" HOT 4
- s390x openjdk8 system tests failing with java.lang.NoSuchMethodError: java/nio/ByteBuffer.position(I)Ljava/nio/ByteBuffer; and java/nio/FloatBuffer.position(I)Ljava/nio/FloatBuffer; HOT 4
- openJcePlusTests_0_FAILED ibm.jceplus.junit.openjceplus.multithread.TestAES_192 Segmentation error vmState=0x00000000
- jdk_lang_j9_0 failure due to java.lang.NullPointerException HOT 1
- OpenJ9 jvm stuck and tools all have no response HOT 4
- prunsrv.exe (Tomcat10.exe) killed when run on top of openj9 HOT 8
- openJcePlusTests_0_FAILED j9vrb.14 ASSERTION FAILED at openj9/runtime/verbose/errormessageframeworkcfr.c:75: ((0 )) HOT 1
- String.replace() errornous compares and replaces one byte instead of two bytes HOT 5
- Fix ValueTypeArrayTestsJIT test failures
- Fix tests in disabled variations of ValueTypeSystemArraycopyTests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openj9.