G1:long remark pauses

Mikael Gerdin mikael.gerdin at oracle.com
Wed May 28 14:35:57 UTC 2014


On Wednesday 28 May 2014 09.13.49 charlie hunt wrote:
> On 05/28/2014 07:43 AM, Mikael Gerdin wrote:
> > On Wednesday 28 May 2014 07.20.59 charlie hunt wrote:
> >> Hi Thomas,
> >> 
> >> About the only thing I can suggest is running the concurrent cycle more
> >> frequently so as to run the remark more frequently.  Perhaps setting
> >> InitiatingHeapOccupancyPercent low enough that the concurrent cycle runs
> >> all the time.
> > 
> > If the weakly reachable objects are reasonably short-lived an increased
> > survivor size and tenuring threshold could allow more weak references to
> > be
> > dealt with by incremental young collections instead.
> 
> Mikael,
> 
> Doesn't the suggestion of tuning survivor size go against the general
> advice of tuning G1, (and more generally tuning young generation sizing
> with G1), since you're essentially overriding the G1 the young GC pause
> time heuristics?

It does, but if the reference processing during remark is a large problem then 
there are only two possible approaches for reducing the time:

1) Allocate less WeakReference objects.
2) Allow WeakReference objects to die before ConcurrentMark has to deal with 
them.

> 
> Fwiw, max tenuring threshold defaults to 15 with G1.
> 
> Perhaps an alternative to explicit sizing of survivor spaces (or young
> gen) would be increasing MaxPauseTimeMillis which should generally
> increase the size of eden and survivor.

That is probably a reasonable approach as well.

/Mikael

> 
> charlie
> 
> > /Mikael
> > 
> >> The remark is showing good parallelism since User=25.82 and real=2.39.
> >> 
> >> I suppose a refactoring of the application to reduce usage of
> >> WeakReferences is another alternative.
> >> 
> >> Btw, does ParallelOld GC or CMS GC show a similar behavior with a high
> >> number of WeakReferences as a GC issue?
> >> 
> >> charlie
> >> 
> >> On 05/28/2014 02:29 AM, Thomas Viessmann wrote:
> >>> Hi Charlie and all,
> >>> 
> >>> you were pretty close. It's the weak references which stops the world,
> >>> not the SoftRefs:
> >>> 
> >>> 248766.241: [GC remark 248766.246: [GC ref-proc248766.246:
> >>> [SoftReference, 184918 refs, 0.1139337 secs]248766.360:
> >>> *[WeakReference, 2104678 refs, 1.9516481 secs]248768.312:*
> >>> [FinalReference, 45 refs, 0.0033450 secs]248768.315:
> >>> [PhantomReference, 106 refs, 0.0029866 secs]248768.318: [JNI Weak
> >>> Reference, 0.0008403 secs], 2.0846210 secs], 2.3878287 secs]
> >>> 
> >>>   [Times: user=25.82 sys=0.87, real=2.39 secs]
> >>> 
> >>> 248768.630: Total time for which application threads were stopped:
> >>> 2.4019674 seconds
> >>> 
> >>> There are also a lot of SoftRefs around but those are not the culprit.
> >>> Varying  -XX:SoftRefLRUPolicyMSPerMB between 100 and 10000 did not
> >>> change anything.
> >>> 
> >>> Any idea how this could be addressed?
> >>> 
> >>> Thanks and Regards
> >>> 
> >>> Thomas
> >>> 
> >>> On 05/16/14 14:40, charlie hunt wrote:
> >>>>  From the sound of what Thomas is describing, this might be one of
> >>>> 
> >>>> those apps that's making heavy use of SoftReferences. Output from
> >>>> -XX:+PrintReferenceGC as Bengt suggested will show if that's the case.
> >>>> 
> >>>> If we see a large number of SoftReferences being processed per GC, we
> >>>> may get further help with tuning the SoftReference reclamation
> >>>> policy, (-XX:SoftRefLRUPolicyMSPerMB).
> >>>> 
> >>>> charlie
> >>>> 
> >>>> On 05/16/2014 07:32 AM, Bengt Rutisson wrote:
> >>>>> Hi Thomas,
> >>>>> 
> >>>>> 
> >>>>> 16 maj 2014 kl. 14:01 skrev Thomas Viessmann
> >>>>> 
> >>>>> <thomas.viessmann at oracle.com <mailto:thomas.viessmann at oracle.com>>:
> >>>>>> Hi Bengt,
> >>>>>> 
> >>>>>> 
> >>>>>> Thanks for confirming. ParallelOld had stop pauses in the range of
> >>>>>> 20 to 30 seconds.
> >>>>>> CMS was a disaster due to extreme fragmentation and high promotion
> >>>>>> rate even with
> >>>>>> huge eden and survivors.
> >>>>> 
> >>>>> Ok, so even with the long remark pauses G1 is performing better than
> >>>>> the other GCs?
> >>>>> 
> >>>>>> There are definitely lots of references. I can find out
> >>>>>> details.
> >>>>> 
> >>>>> Thanks, it would be interesting to get this data.
> >>>>> 
> >>>>> Thanks,
> >>>>> Bengt
> >>>>> 
> >>>>>> Thanks and Regards
> >>>>>> 
> >>>>>> Thomas
> >>>>>> 
> >>>>>> On 05/16/14 13:53, Bengt Rutisson wrote:
> >>>>>>> Hi again Thomas,
> >>>>>>> 
> >>>>>>> On 2014-05-16 13:34, Thomas Viessmann wrote:
> >>>>>>>> Hi Bengt,
> >>>>>>>> 
> >>>>>>>> Sure, the application has lots of objects and references.
> >>>>>>>> Downsizing the application has been tried The heap size of 24 g is
> >>>>>>>> already at minimum. A  smaller heap gave OutOfmemory really quick.
> >>>>>>>> My question was more whether the remark phases could be optimized
> >>>>>>>> further. I assume this is not the case and we have reached the
> >>>>>>>> limitations
> >>>>>>>> of G1, right?
> >>>>>>> 
> >>>>>>> How many reference objects does the application use? Can you run
> >>>>>>> it with -XX:+PrintReferenceGC to see how many there are?
> >>>>>>> 
> >>>>>>> If there are a lot of them I don't think there is much more that
> >>>>>>> can be done for the remark phase. But if there are not that many I
> >>>>>>> guess it means that the remark phase is inefficient.
> >>>>>>> 
> >>>>>>> Have you tried any of the other GCs? How do they behave with this
> >>>>>>> application?
> >>>>>>> 
> >>>>>>> Thanks,
> >>>>>>> Bengt
> >>>>>>> 
> >>>>>>>> Thanks and Regards
> >>>>>>>> 
> >>>>>>>> Thomas
> >>>>>>>> 
> >>>>>>>> On 05/16/14 13:18, Bengt Rutisson wrote:
> >>>>>>>>> Hi Thomas,
> >>>>>>>>> 
> >>>>>>>>> On 2014-05-16 13:10, Thomas Viessmann wrote:
> >>>>>>>>>> Hi Bengt,
> >>>>>>>>>> 
> >>>>>>>>>> Well, that's already done and it did improve things.
> >>>>>>>>>> argv[21]: -XX:+ParallelRefProcEnabled
> >>>>>>>>>> argv[22]: -XX:ParallelGCThreads=48
> >>>>>>>>> 
> >>>>>>>>> Sorry, I missed that.
> >>>>>>>>> 
> >>>>>>>>>> before -XX:+ParallelRefProcEnabled was set the stop times were in
> >>>>>>>>>> the range of 20 to 60 seconds.>>>>>>>
> >>>>>>>>> 
> >>>>>>>>> OK. Glad it helped some. :)
> >>>>>>>>> 
> >>>>>>>>>> The application is a Cacao by Oracle. So they cannot change it.
> >>>>>>>>> 
> >>>>>>>>> Is there some way of reducing the amount of reference objects
> >>>>>>>>> that Cacao uses? Does it have cache sizes or similar that can be
> >>>>>>>>> tuned. With a JFR recording we might be able to figure out where
> >>>>>>>>> the reference objects come from.
> >>>>>>>>> 
> >>>>>>>>> Thanks,
> >>>>>>>>> Bengt
> >>>>>>>>> 
> >>>>>>>>>> Thanks and Regards
> >>>>>>>>>> 
> >>>>>>>>>> Thomas
> >>>>>>>>>> 
> >>>>>>>>>> On 05/16/14 12:58, Bengt Rutisson wrote:
> >>>>>>>>>>> Hi Thomas,
> >>>>>>>>>>> 
> >>>>>>>>>>> It looks like the application is using a lot of Reference
> >>>>>>>>>>> objects. The time spent in remark is dominated by reference
> >>>>>>>>>>> processing. See the attached graph generated from the log file
> >>>>>>>>>>> you sent.
> >>>>>>>>>>> 
> >>>>>>>>>>> You can try to see if adding -XX:+ParallelRefProcEnabled
> >>>>>>>>>>> improves the situation.
> >>>>>>>>>>> 
> >>>>>>>>>>> If the customer is interested in updating their application
> >>>>>>>>>>> they might want to see if they can reduce the number of
> >>>>>>>>>>> java.lang.ref.Reference objects they use.
> >>>>>>>>>>> 
> >>>>>>>>>>> Hths,
> >>>>>>>>>>> Bengt
> >>>>>>>>>>> 
> >>>>>>>>>>> On 2014-05-16 10:26, Thomas Viessmann wrote:
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>>> I've been tuning a Java 7u51, Solaris 10, T4 system with 24G
> >>>>>>>>>>>> heap.
> >>>>>>>>>>>> My customer is not very happy with the remark pauses of  up
> >>>>>>>>>>>> to 2 seconds.
> >>>>>>>>>>>> 
> >>>>>>>>>>>>   -XX:ParallelGCThreads=48 turned out to be the optimum. Here
> >>>>>>>>>>>> 
> >>>>>>>>>>>> is the log file
> >>>>>>>>>>>> which contains the java args at the top:
> >>>>>>>>>>>> 
> >>>>>>>>>>>> http://aubing.de.oracle.com/gclog/gc_log_03052014.log
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Any idea to drive the remark stop times further down?
> >>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Thanks and Regards
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Thomas
> >>>>>>>>>> 
> >>>>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>>>>>> 
> >>>>>>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>>>>>> Geschäftsführere: Juergen Kunz
> >>>>>>>>>> 
> >>>>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr.
> >>>>>>>>>> 30143697
> >>>>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>>>>>> 
> >>>>>>>>>> -----------------------------------------------------------------
> >>>>>>>>>> --
> >>>>>>>>>> -----
> >>>>>>>>>> -----------------------------------------------------------------
> >>>>>>>>>> --
> >>>>>>>>>> -----
> >>>>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
> >>>>>>>>>> is committed to developing practices and products that help
> >>>>>>>>>> protect the environment
> >>>>>>>> 
> >>>>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>>>> 
> >>>>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>>>> Geschäftsführere: Juergen Kunz
> >>>>>>>> 
> >>>>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>>>> 
> >>>>>>>> -------------------------------------------------------------------
> >>>>>>>> --
> >>>>>>>> ---
> >>>>>>>> -------------------------------------------------------------------
> >>>>>>>> --
> >>>>>>>> ---
> >>>>>>>> <mime-attachment.gif> <http://www.oracle.com/commitment> Oracle
> >>>>>>>> is committed to developing practices and products that help
> >>>>>>>> protect the environment
> >>>>>> 
> >>>>>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>>>>> 
> >>>>>> ORACLE Deutschland B.V. & Co. KG
> >>>>>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>>>>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>>>>> Geschäftsführere: Juergen Kunz
> >>>>>> 
> >>>>>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>>>>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>>>>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>>>>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>>>>> 
> >>>>>> ---------------------------------------------------------------------
> >>>>>> --
> >>>>>> -
> >>>>>> ---------------------------------------------------------------------
> >>>>>> --
> >>>>>> -
> >>>>>> <green-for-email-sig_0.gif> <http://www.oracle.com/commitment>
> >>>>>> Oracle is committed to developing practices and products that help
> >>>>>> protect the environment
> >>> 
> >>> ORACLE Deutschland B.V. & Co. KG | Riesstr.25 | D-80992 Muenchen
> >>> 
> >>> ORACLE Deutschland B.V. & Co. KG
> >>> Hauptverwaltung: Riesstr. 25, D-80992 Muenchen
> >>> Registergericht: Amtsgericht Muenchen, HRA 95603
> >>> Geschäftsführere: Juergen Kunz
> >>> 
> >>> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> >>> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> >>> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> >>> Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
> >>> 
> >>> ------------------------------------------------------------------------
> >>> ------------------------------------------------------------------------
> >>> Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
> >>> developing practices and products that help protect the environment



More information about the hotspot-gc-dev mailing list