[G1GC] Evacuation failures with bursts of humongous object allocations

Thomas Schatzl thomas.schatzl at oracle.com
Thu Nov 12 10:13:12 UTC 2020


Hi,

On 11.11.20 23:50, Charlie Gracie wrote:
> Hi Thomas,
> 
> Thanks for the detailed reply.
> 
>>   You probably are missing "... that are short-living" in your
>>   description. Otherwise the suggested workaround does not... work.
> 
> Yes that is correct I should have said short lived humongous object allocations.
> 
>>   Why regular objects too?
> 
> Originally I only added it to the humongous allocation path. Occasionally, the issue
> would still happen if the burst of short-lived humongous allocations finished before
> the free region count dropped below my threshold. Eden regions would continue
> to consume free regions and the following GC encountered To-space exhaustion
> due to 0 free regions. Adding the same check to both slow allocation paths resolved
> the issue and after reviewing it more I believe it needs to be there.

Me too.

> 
>>   Maybe to completely obsolete G1ReservePercent for this purpose?
> 
> Yes I think that a change like this could obsolete G1ReservePercent. I have been
> testing my changes with G1ReservedPercent set to 0 by default for the last few
> days without any issue.

G1ReservePercent is still used by adaptive IHOP for a slightly different 
purpose, so we can't remove it/obsolete right away. But it is good to 
find a better alternative for one of its uses.

> 
>>   This looks like: "do a gc if the amount of free space after evacuating
>>   currently allocated regions is larger than the allocation".
> 
> Yes that is it.
> 
>>   - for the first term, eden regions, g1 already provides more accurate(?)
>>   prediction of survived bytes after evac using survival rate predictors
>>   (see G1SurvRateGroup), one use is in
>>   G1Policy::predict_eden_copy_time_ms(), another in
>>   G1Policy::predict_bytes_to_copy.
> 
> My initial prototype was being very conservative in its decisions to try to avoid the situation
> while I continued the investigation. Improving the calculation will make this change much
> better. Thanks for pointing me in the right direction!
> 
> I am now testing a version that queries eden_surv_rate_group->accum_surv_rate_pred
> using the current count of Eden regions. This is providing very accurate values. Sometimes
> it is a little off because of allocation fragmentation as you explained. To combat this I am
> currently multiplying the result by 1.05 to compensate for the allocation fragmentation
> in the PLABs. The extra 5% could likely be replaced with some historical data based on
> PLAB wasted stats.

There is already a TargetPLABWastePct option that is typically kept very 
well by actual allocation.

There is another complication that measure does not catch, that is 
fragmentation/waste at the end of the last allocation region (for 
survivors). This is due to that regions are required to be of a 
particular type, i.e. you can't have regions of two different types, so 
you need to waste the space at the end of that last region.

This only applies to survivor regions, G1 reuses the last old gen 
allocation region for the next gc (if useful and possible) there. You 
obviously can't do that meaningfully for survivors as they are evacuated 
in whole the next gc again.

> 
> Is this more along the lines of what you were thinking for this calculation?

Yes. See above paragraph for some more thoughts on how to refine this.

> 
>>   Note that G1 does track survival rate for survivor regions too, but it's
>>   not good in my experience - survival rate tracking assumes that objects
>>   within a region are of approximately the same age (the survival rate
>>   prediction mechanism assigns the same "age" to objects allocated in the
>>   same region. This is not the object age in the object headers generally
>>   used!), which the objects in survivor regions tend to simply not be. The
>>   objects in there are typically jumbled together from many different ages
>>   which completely violates that assumption.
> 
> I am currently investigating iterating the survivor regions at the end of a GC
> and calling G1Policy::predict_bytes_to_copy on each of them to estimate
> their survival rate for the next collect. For my tests the result seem accurate
> but I will trust your assessment that it can be off due to the reasons you listed.
> This is definitely an improvement of my initial prototype.

We can start by using that and refine from there.

> 
>>   - potential surviving objects from old region evacuations are missing
>>   completely in the formula. I presume in your case these were not
>>   interesting because (likely) this application mostly does short living
>>   humongous allocations and otherwise keeps a stable old gen?
> 
> You are correct. In my case old space is very stable if you exclude the short-lived
> humongous allocations. A final solution should have included this to be complete,
> but if G1ReservePercent is obsoleted as part of this old regions _need_ to be
> incorporated to make sure no workloads are regressed.
> 
> I believe this could be calculated at the end of a pause (young or mixed), if the next
> GC will be a mixed collect, otherwise it is 0. Do you agree?

Unfortunately, I need to disagree :) Because some survivors will get 
spilled into old all the time (in a non-optimized general application).

However old gen allocation is already tracked per GC as a side-effect of 
the existing PLAB allocation tracking mechanism. See 
G1CollectedHeap::record_obj_copy_mem_stats() about these values in more 
detail.

> 
> I have looked into walking N regions of G1Policy::_collect_set->candidates() and
> calling G1Policy:predict_bytes_to_copy to calculate the amount that may be
> evacuated. I have been using G1Policy::calc_min_old_cset_length() to decide how
> many regions to include in the calculation. Is this a reasonable approach?

For old gen candidates this seems to be a good approach to at least get 
an idea about the spilling. Note that G1Policy:predict_bytes_to_copy() 
just returns HeapRegion::used() for old gen regions, so this will be 
quite conservative. Any improvements to that are welcome, but it will 
work for a first heuristic obviously :)

One could also designate this kind of gc as kind of 
"evacuation-failure-prevention" gc, and use that to direct the old gen 
region selection code in that gc to limit itself to the minimum old gen 
cset length too for the first pass(!) now (I have not thought about 
implications of that - it might not be a good idea because of the 
expected timing of these failures usually being close to mark finish 
anyway and typically the first few regions in the collection set 
candidates are almost empty, so it would be a huge gain to clean out as 
much as possible; but the possibility of the incremental old gen 
evacuation provides a huge safety net here; just initial random thoughts 
without having actually put time into it) - unless we want to add some 
prediction, for example per region-age based, for it.

> 
>>   So basically the intent is to replace G1ReservePercent for this purpose,
>>   making it automatic, which is not a bad idea at all.
>>    
>>   One problem I can see in this situation is that what if that GC does not
>>   free humongous objects memory? Is the resulting behavior better than
>>   before, or in which situation it is(n't)?
> 
> My initial expectation was that if the humongous objects are long lived this change
> does not impact those situations. This is something that I have to think about more
> to come up with a better / complete answer.
> 
>>   And, is there anything that can be done to speed up evacuation failure?
>>   :) Answering my rhetorical question: very likely, see the issues with
>>   evacuation failure collected using the gc-g1-pinned-regions labels
>>   lately [1], in particular JDK-8254739 [2].
> 
> Thanks! Added them to my reading list as I think improving evacuation failure
> would be great even if I wasn't fighting this current issue :)

I just saw that I wasn't specific in the CR what I was thinking about. 
Duh. I updated some CRs a bit, and added more with more ideas to the 
list of those marked with gc-g1-pinned-regions. (If nobody asks about 
it, I'm too lazy to write them down).

Please tell me before starting working on any of those, I might have 
prototype code or additional thoughts about all of them.

> 
>>   So it would be interesting to see the time distribution for evacuation
>>   failure (gc+phases=trace) and occupancy distribution of these failures.
> 
> I will upload some full logs with that option enabled to my GitHub or
> somewhere accessible soon and point you to them. As an FYI I am keeping
> the GitHub branch listed in my first email up to date with my current changes.
> 

Okay. Thanks.

> Thanks a lot for the comments and suggestions!
> Charlie
> 

Thanks,
   Thomas


More information about the hotspot-gc-dev mailing list