G1:long remark pauses

Thomas Viessmann thomas.viessmann at oracle.com
Wed May 28 07:29:33 UTC 2014


Hi Charlie and all,

you were pretty close. It's the weak references which stops the world, 
not the SoftRefs:

248766.241: [GC remark 248766.246: [GC ref-proc248766.246: 
[SoftReference, 184918 refs, 0.1139337 secs]248766.360:
*[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:* 
[FinalReference, 45 refs, 0.0033450 secs]248768.315:
[PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak 
Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
  [Times: user=25.82 sys=0.87, real=2.39 secs]
248768.630: Total time for which application threads were stopped: 
2.4019674 seconds

There are also a lot of SoftRefs around but those are not the culprit.
Varying  -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
change anything.

Any idea how this could be addressed?

Thanks and Regards

Thomas




On 05/16/14 14:40, charlie hunt wrote:
> From the sound of what Thomas is describing, this might be one of 
> those apps that's making heavy use of SoftReferences. Output from 
> -XX:+PrintReferenceGC as Bengt suggested will show if that's the case.
>
> If we see a large number of SoftReferences being processed per GC, we 
> may get further help with tuning the SoftReference reclamation policy, 
> (-XX:SoftRefLRUPolicyMSPerMB).
>
> charlie
>
> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
>>
>> Hi Thomas,
>>
>>
>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann 
>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
>>
>>> Hi Bengt,
>>>
>>>
>>> Thanks for confirming. ParallelOld had stop pauses in the range of 
>>> 20 to 30 seconds.
>>> CMS was a disaster due to extreme fragmentation and high promotion 
>>> rate even with
>>> huge eden and survivors.
>>
>> Ok, so even with the long remark pauses G1 is performing better than 
>> the other GCs?
>>
>>
>>> There are definitely lots of references. I can find out
>>> details.
>>
>> Thanks, it would be interesting to get this data.
>>
>> Thanks,
>> Bengt
>>
>>>
>>> Thanks and Regards
>>>
>>> Thomas
>>>
>>>
>>>
>>> On 05/16/14 13:53, Bengt Rutisson wrote:
>>>>
>>>> Hi again Thomas,
>>>>
>>>>
>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
>>>>> Hi Bengt,
>>>>>
>>>>> Sure, the application has lots of objects and references.
>>>>> Downsizing the application has been tried The heap size of 24 g is
>>>>> already at minimum. A  smaller heap gave OutOfmemory really quick.
>>>>> My question was more whether the remark phases could be optimized
>>>>> further. I assume this is not the case and we have reached the 
>>>>> limitations
>>>>> of G1, right?
>>>>
>>>>
>>>> How many reference objects does the application use? Can you run it 
>>>> with -XX:+PrintReferenceGC to see how many there are?
>>>>
>>>> If there are a lot of them I don't think there is much more that 
>>>> can be done for the remark phase. But if there are not that many I 
>>>> guess it means that the remark phase is inefficient.
>>>>
>>>> Have you tried any of the other GCs? How do they behave with this 
>>>> application?
>>>>
>>>> Thanks,
>>>> Bengt
>>>>
>>>>
>>>>>
>>>>> Thanks and Regards
>>>>>
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
>>>>>>
>>>>>> Hi Thomas,
>>>>>>
>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
>>>>>>> Hi Bengt,
>>>>>>>
>>>>>>> Well, that's already done and it did improve things.
>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
>>>>>>> argv[22]: -XX:ParallelGCThreads=48
>>>>>>
>>>>>> Sorry, I missed that.
>>>>>>
>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times were in the range of 20 to 60 seconds.
>>>>>>
>>>>>> OK. Glad it helped some. :)
>>>>>>
>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
>>>>>>
>>>>>> Is there some way of reducing the amount of reference objects 
>>>>>> that Cacao uses? Does it have cache sizes or similar that can be 
>>>>>> tuned. With a JFR recording we might be able to figure out where 
>>>>>> the reference objects come from.
>>>>>>
>>>>>> Thanks,
>>>>>> Bengt
>>>>>>
>>>>>>> Thanks and Regards
>>>>>>>
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
>>>>>>>>
>>>>>>>> Hi Thomas,
>>>>>>>>
>>>>>>>> It looks like the application is using a lot of Reference 
>>>>>>>> objects. The time spent in remark is dominated by reference 
>>>>>>>> processing. See the attached graph generated from the log file 
>>>>>>>> you sent.
>>>>>>>>
>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled 
>>>>>>>> improves the situation.
>>>>>>>>
>>>>>>>> If the customer is interested in updating their application 
>>>>>>>> they might want to see if they can reduce the number of 
>>>>>>>> java.lang.ref.Reference objects they use.
>>>>>>>>
>>>>>>>> Hths,
>>>>>>>> Bengt
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G 
>>>>>>>>> heap.
>>>>>>>>> My customer is not very happy with the remark pauses of  up to 
>>>>>>>>> 2 seconds.
>>>>>>>>>  -XX:ParallelGCThreads=48 turned out to be the optimum. Here 
>>>>>>>>> is the log file
>>>>>>>>> which contains the java args at the top:
>>>>>>>>>
>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
>>>>>>>>>
>>>>>>>>> Any idea to drive the remark stop times further down?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards
>>>>>>>>>
>>>>>>>>> Thomas
>>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> <mime-attachment.gif> <http://www.oracle.com>
>>>>>>> THOMAS VIESSMANN | Senior Principal Technical Support Engineer - 
>>>>>>> Java
>>>>>>> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
>>>>>>> <tel:+491743005467>
>>>>>>> Oracle Customer Technical Support - Java
>>>>>>>
>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>>>
>>>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>>>> Geschäftsführere: Juergen Kunz
>>>>>>>
>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>> ------------------------------------------------------------------------
>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle 
>>>>>>> is committed to developing practices and products that help 
>>>>>>> protect the environment
>>>>>>
>>>>>
>>>>> -- 
>>>>> <mime-attachment.gif> <http://www.oracle.com>
>>>>> THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
>>>>> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
>>>>> <tel:+491743005467>
>>>>> Oracle Customer Technical Support - Java
>>>>>
>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>>>
>>>>> ORACLE Deutschland B.V. & Co. KG
>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>>>> Geschäftsführere: Juergen Kunz
>>>>>
>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> ------------------------------------------------------------------------
>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle is 
>>>>> committed to developing practices and products that help protect 
>>>>> the environment
>>>>
>>>
>>> -- 
>>> <oracle_sig_logo.gif> <http://www.oracle.com>
>>> THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
>>> Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
>>> <tel:+491743005467>
>>> Oracle Customer Technical Support - Java
>>>
>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
>>>
>>> ORACLE Deutschland B.V. & Co. KG
>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
>>> Registergericht: Amtsgericht Muenchen, HRA 95603
>>> Geschäftsführere: Juergen Kunz
>>>
>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
>>>
>>> ------------------------------------------------------------------------
>>> ------------------------------------------------------------------------
>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment> 
>>> Oracle is committed to developing practices and products that help 
>>> protect the environment
>

-- 
Oracle <http://www.oracle.com>
THOMAS VIESSMANN | Senior Principal Technical Support Engineer - Java
Phone: +498914302496 <tel:+49814302496> | Mobile: +491743005467 
<tel:+491743005467>
Oracle Customer Technical Support - Java

ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
Registergericht: Amtsgericht Muenchen, HRA 95603
Geschäftsführere: Juergen Kunz

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher

------------------------------------------------------------------------
------------------------------------------------------------------------
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to 
developing practices and products that help protect the environment
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20140528/6c489327/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: oracle_sig_logo.gif
Type: image/gif
Size: 658 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20140528/6c489327/oracle_sig_logo.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: green-for-email-sig_0.gif
Type: image/gif
Size: 356 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20140528/6c489327/green-for-email-sig_0.gif>


More information about the hotspot-gc-dev mailing list