RFR(M/L): 7176479: G1: JVM crashes on T5-8 system with 1.5 TB heap

John Cuthbertson john.cuthbertson at oracle.com
Tue Jan 15 23:31:10 UTC 2013

Hi Everyone,

Can I have a couple of people look over the changes for this CR - the 
webrev can be found at: http://cr.openjdk.java.net/~johnc/7176479/webrev.0/

The issue here was that we were encoding the card index into the card 
counts table entries along with the GC number so that we could determine 
if the count associated with was valid. We had a check to ensure that 
the maximum card index could be encoded in an int. With such large heap 
size - the number of cards could not be encoded and so the check failed.

The previous mechanism was an attempt to solve the problem of one thread 
arriving late to the actual GC work. The thread in question was being 
held up zeroing the card counts table at the start of the GC. The card 
counts table is used to determine which cards are being refined 
frequently. Once a card has been refined frequently enough, further 
refinements of that card are delayed by placing the card into a fixed 
size evicting table - the hot card cache. The card would then be refined 
when it was evicted from the hot card cache or when the cache was 
drained during the next GC.

To solve the problem of zeroing we added an epoch (GC number) to the 
entries in the counts table and, eliminate the increase in footprint, we 
made the counts table into a cache which would expand if needed. This 
approach had some negatives: we might have to refine two cards during a 
single refinement operation, hashing the card, and performing CAS 
operations increasing the overhead of concurrent refinement. Also 
expanding the counts table during a GC incurred a penalty.

This approach also limited the heap size to just under 1TB - which the 
systems team ran into.

The new approach effectively undoes the previous mechanism and 
re-simplifies the card counts table.

Summary of Changes:
The hot card cache and card counts table have been moved from the 
concurrent refinement code into their own files.

The hot card cache can now exist independently of whether the counts 
table exists. In this case refining a card once adds it to the hot card 
cache, i.e. all cards are treated as 'hot'.

The interface to the hot card cache has been simplified - a simple query 
and a simple drain routine. This simplifies the calling code in 
g1RemSet.cpp and results in up to only a single card being refined for 
every call to "refine_card" instead of possibly two. This should reduce 
the overhead of concurrent refinement.

The number of cards that the hot card cache can hold before cards start 
getting evicted is controlled by the flag G1ConcRSLogCacheSize, which is 
now product flag. The default value is 10 giving a hot card cache that 
can hold 1K cards.

The card counts table has been greatly simplified. It is a simple array 
of counts how many times a card has been refined. The space for the 
table is now allocated from virtual memory instead of C heap. The space 
for the table is committed when the heap is initially committed and the 
spans the committed size of the heap. When the committed size of the 
heap is expanded, the counts table is also expanded to cover the newly 
expanded heap. If we fail to commit the memory for the counts table, 
cards that map to the uncommitted space will be treated as cold, i.e. 
they will be refined immediately. Having a simpler counts table also 
should reduce the overhead of concurrent refinement (there is no need to 
hash the card index and there are no CAS operations) Having a simpler 
interface will allow us to change the underlying data structure to an 
alternative that's perhaps more sparse in the future.

During an incremental GC we no longer zero the entire counts table. We 
now zero the cards spanned by a region when the region is freed (i.e. 
when we free the collection set at the end of a GC and when we free 
regions at the end of a cleanup).  If a card was "hot" before a GC then 
we will consider it hot after the GC and the first refinement after the 
GC will insert the card into the hot card cache. Furthermore, since we 
don't refine cards in young regions, we only need to clear the counts 
associated with cards spanned by non-young regions.

During a full GC we still discard the entries in the hot card cache and 
zero the counts for all the cards in the heap.

GC Test suite with MaxTenuringThreshold=0 (to increase the amount of 
refinement) and a low IHOP value (to force cleanups).
SPECjbb2005 with a 1.5TB heap size and 256GB young size, 
MaxTenuringThreshold=0 and a low IHOP value (1%). The systems team are 
continuing to test with very large heaps.

More information about the hotspot-gc-dev mailing list