G1:long remark pauses

Thomas Viessmann thomas.viessmann at oracle.com
Wed May 28 14:36:42 UTC 2014


Hi all,


FYI: this application has lots of long living objects. I quickly gave up 
CMS due to high promotion rate
and extreme fragmentation. Even with huge eden and survivors configured. 
I have already set
InitiatingHeapOccupancyPercent=45 while heap is around 60% full on average.
Well, redesigning an application with fewer weak refs is not really a 
short term  option after all ;-)


Thanks and Regards

Thomas





On 05/28/14 16:13, charlie hunt wrote:
> On 05/28/2014 07:43 AM, Mikael Gerdin wrote:
>> On Wednesday 28 May 2014 07.20.59 charlie hunt wrote:
>>> Hi Thomas,
>>>
>>> About the only thing I can suggest is running the concurrent cycle more
>>> frequently so as to run the remark more frequently.  Perhaps setting
>>> InitiatingHeapOccupancyPercent low enough that the concurrent cycle 
>>> runs
>>> all the time.
>> If the weakly reachable objects are reasonably short-lived an increased
>> survivor size and tenuring threshold could allow more weak references 
>> to be
>> dealt with by incremental young collections instead.
> Mikael,
>
> Doesn't the suggestion of tuning survivor size go against the general 
> advice of tuning G1, (and more generally tuning young generation 
> sizing with G1), since you're essentially overriding the G1 the young 
> GC pause time heuristics?
>
> Fwiw, max tenuring threshold defaults to 15 with G1.
>
> Perhaps an alternative to explicit sizing of survivor spaces (or young 
> gen) would be increasing MaxPauseTimeMillis which should generally 
> increase the size of eden and survivor.
>
> charlie
>
>>
>> /Mikael
>>
>>> The remark is showing good parallelism since User=25.82 and real=2.39.
>>>
>>> I suppose a refactoring of the application to reduce usage of
>>> WeakReferences is another alternative.
>>>
>>> Btw, does ParallelOld GC or CMS GC show a similar behavior with a high
>>> number of WeakReferences as a GC issue?
>>>
>>> charlie
>>>
>>> On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
>>>> Hi Charlie and all,
>>>>
>>>> you were pretty close. It's the weak references which stops the world,
>>>> not the SoftRefs:
>>>>
>>>> 248766.241: [GC remark 248766.246: [GC ref-proc248766.246:
>>>> [SoftReference, 184918 refs, 0.1139337 secs]248766.360:
>>>> *[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:*
>>>> [FinalReference, 45 refs, 0.0033450 secs]248768.315:
>>>> [PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak
>>>> Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
>>>>
>>>>   [Times: user=25.82 sys=0.87, real=2.39 secs]
>>>>
>>>> 248768.630: Total time for which application threads were stopped:
>>>> 2.4019674 seconds
>>>>
>>>> There are also a lot of SoftRefs around but those are not the culprit.
>>>> Varying  -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
>>>> change anything.
>>>>
>>>> Any idea how this could be addressed?
>>>>
>>>> Thanks and Regards
>>>>
>>>> Thomas
>>>>
>>>> On 05/16/14 14:40, charlie hunt wrote:
>>>>>  From the sound of what Thomas is describing, this might be one of
>>>>> those apps that's making heavy use of SoftReferences. Output from
>>>>> -XX:+PrintReferenceGC as Bengt suggested will show if that's the 
>>>>> case.
>>>>>
>>>>> If we see a large number of SoftReferences being processed per GC, we
>>>>> may get further help with tuning the SoftReference reclamation
>>>>> policy, (-XX:SoftRefLRUPolicyMSPerMB).
>>>>>
>>>>> charlie
>>>>>
>>>>> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
>>>>>> Hi Thomas,
>>>>>>
>>>>>>
>>>>>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann
>>>>>>
>>>>>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
>>>>>>> Hi Bengt,
>>>>>>>
>>>>>>>
>>>>>>> Thanks for confirming. ParallelOld had stop pauses in the range of
>>>>>>> 20 to 30 seconds.
>>>>>>> CMS was a disaster due to extreme fragmentation and high promotion
>>>>>>> rate even with
>>>>>>> huge eden and survivors.
>>>>>> Ok, so even with the long remark pauses G1 is performing better than
>>>>>> the other GCs?
>>>>>>
>>>>>>> There are definitely lots of references. I can find out
>>>>>>> details.
>>>>>> Thanks, it would be interesting to get this data.
>>>>>>
>>>>>> Thanks,
>>>>>> Bengt
>>>>>>
>>>>>>> Thanks and Regards
>>>>>>>
>>>>>>> Thomas
>>>>>>>
>>>>>>> On 05/16/14 13:53, Bengt Rutisson wrote:
>>>>>>>> Hi again Thomas,
>>>>>>>>
>>>>>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
>>>>>>>>> Hi Bengt,
>>>>>>>>>
>>>>>>>>> Sure, the application has lots of objects and references.
>>>>>>>>> Downsizing the application has been tried The heap size of 24 
>>>>>>>>> g is
>>>>>>>>> already at minimum. A  smaller heap gave OutOfmemory really 
>>>>>>>>> quick.
>>>>>>>>> My question was more whether the remark phases could be optimized
>>>>>>>>> further. I assume this is not the case and we have reached the
>>>>>>>>> limitations
>>>>>>>>> of G1, right?
>>>>>>>> How many reference objects does the application use? Can you run
>>>>>>>> it with -XX:+PrintReferenceGC to see how many there are?
>>>>>>>>
>>>>>>>> If there are a lot of them I don't think there is much more that
>>>>>>>> can be done for the remark phase. But if there are not that many I
>>>>>>>> guess it means that the remark phase is inefficient.
>>>>>>>>
>>>>>>>> Have you tried any of the other GCs? How do they behave with this
>>>>>>>> application?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bengt
>>>>>>>>
>>>>>>>>> Thanks and Regards
>>>>>>>>>
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
>>>>>>>>>> Hi Thomas,
>>>>>>>>>>
>>>>>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
>>>>>>>>>>> Hi Bengt,
>>>>>>>>>>>
>>>>>>>>>>> Well, that's already done and it did improve things.
>>>>>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
>>>>>>>>>>> argv[22]: -XX:ParallelGCThreads=48
>>>>>>>>>> Sorry, I missed that.
>>>>>>>>>>
>>>>>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times 
>>>>>>>>>>> were in
>>>>>>>>>>> the range of 20 to 60 seconds.>>>>>>>
>>>>>>>>>> OK. Glad it helped some. :)
>>>>>>>>>>
>>>>>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
>>>>>>>>>> Is there some way of reducing the amount of reference objects
>>>>>>>>>> that Cacao uses? Does it have cache sizes or similar that can be
>>>>>>>>>> tuned. With a JFR recording we might be able to figure out where
>>>>>>>>>> the reference objects come from.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bengt
>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
>>>>>>>>>>>> Hi Thomas,
>>>>>>>>>>>>
>>>>>>>>>>>> It looks like the application is using a lot of Reference
>>>>>>>>>>>> objects. The time spent in remark is dominated by reference
>>>>>>>>>>>> processing. See the attached graph generated from the log file
>>>>>>>>>>>> you sent.
>>>>>>>>>>>>
>>>>>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled
>>>>>>>>>>>> improves the situation.
>>>>>>>>>>>>
>>>>>>>>>>>> If the customer is interested in updating their application
>>>>>>>>>>>> they might want to see if they can reduce the number of
>>>>>>>>>>>> java.lang.ref.Reference objects they use.
>>>>>>>>>>>>
>>>>>>>>>>>> Hths,
>>>>>>>>>>>> Bengt
>>>>>>>>>>>>
>>>>>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G
>>>>>>>>>>>>> heap.
>>>>>>>>>>>>> My customer is not very happy with the remark pauses of  up
>>>>>>>>>>>>> to 2 seconds.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   -XX:ParallelGCThreads=48 turned out to be the optimum. Here
>>>>>>>>>>>>>
>>>>>>>>>>>>> is the log file
>>>>>>>>>>>>> which contains the java args at the top:
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any idea to drive the remark stop times further down?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thomas
>>>>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 
>>>>>>>>>>> Muenchen
>>>>>>>>>>>
>>>>>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>>>>>
>>>>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 
>>>>>>>>>>> 30143697
>>>>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val 
>>>>>>>>>>> Maher
>>>>>>>>>>>
>>>>>>>>>>> ------------------------------------------------------------------- 
>>>>>>>>>>>
>>>>>>>>>>> -----
>>>>>>>>>>> ------------------------------------------------------------------- 
>>>>>>>>>>>
>>>>>>>>>>> -----
>>>>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
>>>>>>>>>>> is committed to developing practices and products that help
>>>>>>>>>>> protect the environment
>>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>>>>
>>>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>>>
>>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 
>>>>>>>>> 30143697
>>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------------- 
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> --------------------------------------------------------------------- 
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
>>>>>>>>> is committed to developing practices and products that help
>>>>>>>>> protect the environment
>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>>
>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>
>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>>
>>>>>>> ----------------------------------------------------------------------- 
>>>>>>>
>>>>>>> -
>>>>>>> ----------------------------------------------------------------------- 
>>>>>>>
>>>>>>> -
>>>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
>>>>>>> Oracle is committed to developing practices and products that help
>>>>>>> protect the environment
>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>
>>>> ORACLE Deutschland B.V. & Co. KG
>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>> Geschäftsführere: Juergen Kunz
>>>>
>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>
>>>> ------------------------------------------------------------------------ 
>>>>
>>>> ------------------------------------------------------------------------ 
>>>>
>>>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
>>>> developing practices and products that help protect the environment
>

-- 
Oracle <http://www.oracle.com>
THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
<tel:+491743005467>
Oracle Customer Technical Support - Java

ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
Registergericht: Amtsgericht Muenchen, HRA 95603
Geschäftsführere: Juergen Kunz

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher

------------------------------------------------------------------------
------------------------------------------------------------------------
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20140528/13b2a29d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20140528/13b2a29d/oracle_sig_logo.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20140528/13b2a29d/green-for-email-sig_0.gif>


More information about the hotspot-gc-dev mailing list