G1 discovers same garbage again?
java at java4.info
Wed Nov 30 00:35:45 PST 2011
first of all thank you for your answere.
Am 29.11.2011 18:29, schrieb Tony Printezis:
> Hi Florian,
> See inline.
> On 11/28/2011 2:18 PM, Florian Binder wrote:
>> Hi everybody,
>> I have a java application with 20gb (large-table) memory and using the
>> g1 garbage collector.
> Quick clarification: I saw that you use a 20G heap from the parameters
> you showed below. Do you know what's your live data size?
At this time I have about 14gb live data but it is growing day by day.
>> The application calculates the whole time with 10 threads some ratios
>> (high cpu load). This is done without producing any garbage. About two
>> times a minute a request is sent which produce a littlebit garbage.
>> Since we are working with realtime data we are interested in very short
>> stop-the-world pauses. Therefore we have used the CMS gc in the past
>> until we have got problems with fragmentation now.
> Since you don't produce much garbage how come you have fragmentation?
> Do you keep the results for all the requests you serve?
This data is hold one day and every night it is droped and reinitialized.
We have a lot of different server with big memory and have had problems
with fragmentation on few of them. This was the cause I am experiencing
with g1 in general. I am not sure if we had fragmentation on this one.
Today I tried the g1 with another server which surely have had a problem
with fragmented heap, but this one did not start wit g1. I got several
different exceptions (NoClassDefFound, NullPointerException or even a
jvm-crash ;-)). But I think I will write you another email especially
for this, because it is started with a lot of special parameters (e.g.
-Xms39G -Xmx39G -XX:+UseCompressedOops -XX:ObjectAlignmentInBytes=16
>> Therefore I am trying the g1.
>> This seemed to work very well at first. The stw-pauses were, except the
>> cleanup pause,
> Out of curiosity: how long are the cleanup pauses?
I think they were about 150ms. This is acceptable for me, but in
proportion to the garbage-collection of 30ms it is very long and
therefore I was wondering.
>> very short. This yields me to my first question:
>> Is this normal and are there any parameters to influence the
> I don't think there's much you can do in the app to influence the
> cleanup duration. During this pause we do some, ahem, cleanup of our
> data structures and for large heaps I have also seen the cleanup
> pauses to take longer than I thought they would take. I know this is
> not going to help you in the short term but we have plans to do the
> cleanup work concurrently (or at least mostly-concurrently) in the
Sounds good :-)
>> I thought this phase should be short because there is
>> just finished the counting, the role of the bitmaps is switched and the
>> next possible garbage regions are detemined. All things, which should be
>> very fast. So what is taking the time?
> Most likely, the remembered set scrubbing phase...
>> The second cause for my email is the crazy behaviour after a few hours:
>> After the startup of the server it uses about 13.5 gb old-gen memory and
>> generates very slowly eden-garbage. Since the new allocated memory is
>> mostly garbage the (young) garbage collections are very fast and g1
>> decides to grow up the eden space. This works 4 times until eden space
>> has more than about 3.5 gb memory. After this the gc is making much more
>> collections and while the collections it discovers new garbage (probably
>> the old one again).
> I'm not quite sure what you mean by "it discovers new garbage". For
> young GCs, G1 (and our other GCs) will reclaim any young objects that
> will discover to be dead (more accurately: that it will not discover
> to be live).
>> Eden memory usage jumps between 0 and 3.5gb even
>> though I am sure the java-application is not making more than before.
> Well, that's not good. :-) Can you try to explicitly set the young gen
> size with -Xmn3g say, to see what happens?
With "it discovers new garbage" I mean that during the garbage
collection the eden space usage jumps up to 3gb. Then it cleans up the
whole garbage (eden usage is 0) and a few seconds later the eden usage
jumps again up. You can see this in the 1h eden-space snapshot:
Since the jumps are betweend 0 and the last max eden usage (of about
3.5gb) I assume that it discovers the same garbage, it cleaned up the
last time, and collects it again. I am sure the application is not
making more garbage than the time before. Have you ever heared of
problems like this?
After I have written the last email, I have seen that it has calm itself
after a few hours. But it is nevertheless very curious and produces a
lot of unnecessary pauses.
>> assume that it runs during a collection in the old garbage and collects
>> it again. Is this possible? Or is there an overflow since eden space
>> uses more than 3.5 gb?
>> Thanks and regards,
>> Some useful information:
>> $ java -version
>> java version "1.6.0_29"
>> Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
>> Startup Parameters:
>> -Xms20g -Xmx20g
>> -verbose:gc \
>> -XX:+UnlockExperimentalVMOptions \
>> -XX:+UseG1GC \
>> -XX:+PrintGCDetails \
>> -XX:+PrintGCDateStamps \
>> -XX:+UseLargePages \
>> -XX:+PrintFlagsFinal \
>> -XX:-TraceClassUnloading \
>> $ cat /proc/meminfo | grep Huge
>> HugePages_Total: 11264
>> HugePages_Free: 1015
>> HugePages_Rsvd: 32
>> Hugepagesize: 2048 kB
>> A few screen-shots of the jconsole memory-view:
>> The sysout end syserr logfile with the gc logging and PrintFinalFlags:
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
More information about the hotspot-gc-use