Code Monkey home page Code Monkey logo

Comments (4)

pshipton avatar pshipton commented on July 26, 2024 1

@dmitripivkine pls take a look

from openj9.

dmitripivkine avatar dmitripivkine commented on July 26, 2024

Adding this item to GC queue.
OOM is triggered by Excessive GC condition. However there are 673 free regions (of 2022 total), 1m each.
This does not look right, we need to look closer.

from openj9.

dmitripivkine avatar dmitripivkine commented on July 26, 2024

There is the problem with Balanced GC - calculation of current Eden size converged to minimum value of two regions and got stuck there. This causes PGCs triggered by Allocation Failure almost back to back up to 95% time GC utilization and Excessive GC condition latch eventually. We are investigating.

from openj9.

amicic avatar amicic commented on July 26, 2024

Some initial observations.

We seem to be constantly doing PGCs, and that's because Eden is only 2MBs (2 regions of 1MB). We should be expanding the Eden since the heap is 2GB and fully expanded. But for some reason we don't expand.
The core logic for Eden expansion (for the case of fully expanded heap) is in calculateRecommendedEdenChangeForExpandedHeap, which has some non-trivai calculatinos. Unfortuantelly a couple of trace point in that method did not show up in the dump (even though one of them is Level 1!?)

However, there are quite a few relevant parameters in SchedulingDelegate, and most of them look reasonable, except maybe _estimatedFreeTenure of about 8MB, which seems rather low (trace points report around 700MB)

> !mm_schedulingdelegate 0x00007FFF90069338 | grep _estimatedFreeTenure
	0x100: U64 _estimatedFreeTenure = 0x000000000083F133 (8646963) 	
	
20:33:12.362818623  0x0 j9mm.346             Event       PGC end: workstackoverflow=0 overflowcount=0 heapFree=715128832/2120220672
20:33:12.365062888  0x0 j9mm.345             Event       PGC start: globalcount=4335 nextGMPIncrement=0
20:33:12.442877284  0x0 j9mm.346             Event       PGC end: workstackoverflow=0 overflowcount=0 heapFree=714080256/2120220672
20:33:12.444926051  0x0 j9mm.345             Event       PGC start: globalcount=4336 nextGMPIncrement=0

GC does observe high PGC over head (more like 84-89%, but still very high and we should expand Eden)

> !mm_schedulingdelegate 0x00007FFF90069338 | grep _historicalPartialGCTime
	0x128: U64 _historicalPartialGCTime = 0x0000000000000043 (67)
> !mm_schedulingdelegate 0x00007FFF90069338 | grep _averagePgcInterval
	0x180: U64 _averagePgcInterval = 0x00000000000129ED (76269)
	
(lldb) p 67 * 1000.0 / 76269
(double) 0.87846962724042532
          
> !mm_schedulingdelegate 0x00007FFF90069338 | grep _partialGcOverhead
	0x120: double _partialGcOverhead = 0.8466813906537427

Evidence that Eden is stuck at 2 regions:

> !mm_schedulingdelegate 0x00007FFF90069338 | grep -i edenregion
	0x30: U64 _idealEdenRegionCount = 0x0000000000000002 (2)
	0x38: U64 _minimumEdenRegionCount = 0x0000000000000001 (1)
	0x40: U64 _edenRegionCount = 0x0000000000000002 (2)
	0x108: U64 _maxEdenRegionCount = 0x00000000000005EC (1516)
	0x110: U64 _minEdenRegionCount = 0x0000000000000002 (2)

Evidence that heap is fully expanded:

> !MM_GCExtensions 0x00007FFF90059500 |grep memoryMax
	0x5fd8: U64 memoryMax = 0x000000007E600000 (2120220672)

> !mm_schedulingdelegate 0x00007FFF90069338 | grep -i _numberOfHeapRegions
	0x58: U64 _numberOfHeapRegions = 0x00000000000007E6 (2022)

(lldb) p 2022 * 1024 * 1024
(int) 2120220672

from openj9.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.