From Jon.Masamitsu at Sun.COM Mon Mar 3 09:23:52 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Mon, 03 Mar 2008 09:23:52 -0800 Subject: no-administration heap? In-Reply-To: References: <87d4qgrd69.fsf@mid.deneb.enyo.de> <47C739B1.3010605@Sun.COM> Message-ID: <47CC3428.2060902@Sun.COM> Colin Walters wrote: > On Thu, Feb 28, 2008 at 5:46 PM, Peter B. Kessler wrote: >> Nothing is "too invasive"! We have a sketch of a "chunked heap"; >> it's just a lot of work. Things like the card table that used to >> be indexed by offsets in the heap become part of each heap chunk >> and are indexed by an offset in the chunk. A lot of the underlying >> data structures have to be rewritten, but that's why they call it >> software. We would *love* to have that kind of heap. > > Great to hear there's work in this area! I'll be keeping an eye on > the progress; I'm planning to convert a CPython application to > OpenJDK+Jython, and this issue is one thing that would be nice to have > solved. > Just to be clear, we would like to have a chunked heap, but we do not currently have anyone working on it. I don't think it's part of our plans for this year. Anyone, please correct me if I've misspoken. From Peter.Kessler at Sun.COM Wed Mar 5 09:56:38 2008 From: Peter.Kessler at Sun.COM (Peter B. Kessler) Date: Wed, 05 Mar 2008 09:56:38 -0800 Subject: advice on tuning JVM In-Reply-To: References: Message-ID: <47CEDED6.6040805@Sun.COM> Shaffer, Kurt wrote: > Hello, > ... > 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > ... > java version "1.4.2_13" > ... -XX:+UseParallelGC -XX:ParallelGCThreads=20 ... > > Application is PTC Windchill > > > > We have seen very slow response times from the users standpoint, when they login it takes up to 15 seconds to render the display, when they are navigating through different views in their browser it can take 40 seconds or longer to display pages. > > > > Thankx, > > Kurt Try turning the number of ParallelGCThreads down to match the number of CPU's on your box. On a 4-cpu box I would start out with -XX:ParallelGCThreads=4, but if I saw a large variance in the times for minor collections, I might try -XX:ParallelGCThreads=3 to leave one CPU for running other things while the JVM was collecting. If you typically run more than one such JVM on the box, you might consider -XX:ParallelGCThreads=2 to be polite. Note that in 1.4.2_13 only the minor (young generation) collections are done on many processors; full collections are still serial collections. If your logs show those collections to be the problem, you might want to try the mostly-concurrent collector (-XX:+UseConcMarkSweepGC) instead, but that will require different tunings. But, as Jon said: mostly we need to see your logs to understand what's not working. It would also help if you could say what kinds of pauses you think are too long. Clearly 15 or 40 second pauses are unacceptable for interactive applications, but, for example, would occasional 1 or 2 seconds pauses be acceptable? ... peter _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Kurt_Shaffer at steris.com Wed Mar 5 08:23:19 2008 From: Kurt_Shaffer at steris.com (Shaffer, Kurt) Date: Wed, 5 Mar 2008 11:23:19 -0500 Subject: Tuning advice Message-ID: Hello, We are looking for some tuning recommendations for the following environment. I am not very familiar with Java so I am seeking any help. > 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > > > > java version "1.4.2_13" > > Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06) > > Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) > > > > overridable="true" value="$(wt.java.cmd.quoted) -server -classpath > $(wt.java.clas > > spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient > -DHTTPClient.disableKeepAlives\=true > -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 > > 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC > -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails > -Xloggc:$(wt.logs.di > > r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> > > > > Application is PTC Windchill > > > > We have seen very slow response times from the users standpoint, when > they login it takes up to 15 seconds to render the display, when they > are navigating through different views in their browser it can take 40 > seconds or longer to display pages. > 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] PrintGCStats output what count total mean max stddev gen0(s) 1282 776.553 0.60574 3.605 0.3765 gen1t(s) 823 1643.837 1.99737 11.139 1.0934 GC(s) 2105 2420.391 1.14983 11.139 1.0074 alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 MB/s alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 MB/s alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 MB/s promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 MB/s promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 MB/s gc_seq_load = 9681.563 s / 227507.228 s = 4.255% gc_conc_load = 0.000 s / 227507.228 s = 0.000% gc_tot_load = 9681.563 s / 227507.228 s = 4.255% > > Thankx Kurt Shaffer UNIX Architect IT Enterprise Operations STERIS Corporation 440-392-7089 Internal Sites * UNIX Operations * STERIS Blogs * STERIS Wiki * STERIS Java Instant Messanger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20080305/2a0fc0ca/attachment-0001.html -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Wed Mar 5 11:37:13 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Wed, 05 Mar 2008 11:37:13 -0800 Subject: Tuning advice In-Reply-To: References: Message-ID: <47CEF669.2010707@Sun.COM> Kurt, The full GC's in the log fragment that you sent are at very regular intervals so there might be something in the system that is invoking System.gc(). Full GC's normally happen when the heap is full and the logs show that the heap is not yet full when the full GC's happen. If the frequency of the full GC's is a problem, you can use -XX:+DisableExplicitGC to effectively turn off System.gc(). Other than that, it does not look like GC is a problem. The full GC's are all under 6 seconds and are about 60 seconds apart so I don't see how that can be causing the rendering of a page to take 40 seconds. Are there other parts of the gc log that show longer and/or more frequent GC pauses? Jon Shaffer, Kurt wrote: > Hello, > > > > We are looking for some tuning recommendations for the following > environment. I am not very familiar with Java so I am seeking any help. > > > >> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > >> > >> > >> > >> java version "1.4.2_13" > >> > >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06) > >> > >> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) > >> > >> > >> > >> >> overridable="true" value="$(wt.java.cmd.quoted) -server -classpath > >> $(wt.java.clas > >> > >> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient > >> -DHTTPClient.disableKeepAlives\=true > >> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 > >> > >> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC > >> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails > >> -Xloggc:$(wt.logs.di > >> > >> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> > >> > >> > >> > >> Application is PTC Windchill > >> > >> > >> > >> We have seen very slow response times from the users standpoint, when > >> they login it takes up to 15 seconds to render the display, when they > >> are navigating through different views in their browser it can take 40 > >> seconds or longer to display pages. > >> > > 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] > > 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] > > 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] > > 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] > > 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] > > 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] > > 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] > > 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] > > 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] > > 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] > > 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] > > 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] > > 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] > > 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] > > 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] > > 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] > > 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] > > 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] > > 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] > > 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] > > 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] > > 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] > > 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] > > 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] > > 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] > > 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] > > 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] > > 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] > > 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] > > 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] > > 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] > > 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] > > 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] > > 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] > > 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] > > > > PrintGCStats output > > what count total mean max stddev > > gen0(s) 1282 776.553 0.60574 3.605 0.3765 > > gen1t(s) 823 1643.837 1.99737 11.139 1.0934 > > GC(s) 2105 2420.391 1.14983 11.139 1.0074 > > alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 > > promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 > > > > alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 MB/s > > alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 MB/s > > alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 MB/s > > promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 MB/s > > promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 MB/s > > gc_seq_load = 9681.563 s / 227507.228 s = 4.255% > > gc_conc_load = 0.000 s / 227507.228 s = 0.000% > > gc_tot_load = 9681.563 s / 227507.228 s = 4.255% > >> > >> Thankx > > > > /Kurt Shaffer/ > UNIX Architect > IT Enterprise Operations > STERIS Corporation > 440-392-7089 > > Internal Sites > > * UNIX Operations > > * STERIS Blogs > * STERIS Wiki > * STERIS Java Instant Messanger > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From tony.printezis at sun.com Wed Mar 5 11:37:59 2008 From: tony.printezis at sun.com (Tony Printezis) Date: Wed, 05 Mar 2008 14:37:59 -0500 Subject: Tuning advice In-Reply-To: <47CEF669.2010707@Sun.COM> References: <47CEF669.2010707@Sun.COM> Message-ID: <47CEF697.30902@sun.com> Kurt, Do you use RMI? By default, RMI calls Full GC every 60 seconds for the correct operation of its distributed GC algorithm. The default behavior has been changed in 6.0 to be something less frequent (once an hour I believe...). Tony Jon Masamitsu wrote: > Kurt, > > The full GC's in the log fragment that you sent are at > very regular intervals so there might be something > in the system that is invoking System.gc(). Full > GC's normally happen when the heap is full and > the logs show that the heap is not yet full when > the full GC's happen. If the frequency of the full > GC's is a problem, you can use -XX:+DisableExplicitGC > to effectively turn off System.gc(). > > Other than that, it does not look like GC is a problem. The > full GC's are all under 6 seconds and are about 60 seconds > apart so I don't see how that can be causing the rendering > of a page to take 40 seconds. Are there other parts of the > gc log that show longer and/or more frequent GC pauses? > > Jon > > Shaffer, Kurt wrote: > >> Hello, >> >> >> >> We are looking for some tuning recommendations for the following >> environment. I am not very familiar with Java so I am seeking any help. >> >> >> >> >>> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory >>> >>> java version "1.4.2_13" >>> >>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06) >>> >>> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) >>> >>> >> >>> overridable="true" value="$(wt.java.cmd.quoted) -server -classpath >>> >>> $(wt.java.clas >>> >>> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient >>> >>> -DHTTPClient.disableKeepAlives\=true >>> >>> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 >>> >>> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC >>> >>> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails >>> >>> -Xloggc:$(wt.logs.di >>> >>> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> >>> >>> Application is PTC Windchill >>> >>> We have seen very slow response times from the users standpoint, when >>> >>> they login it takes up to 15 seconds to render the display, when they >>> >>> are navigating through different views in their browser it can take 40 >>> >>> seconds or longer to display pages. >>> >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] >> >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] >> >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] >> >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] >> >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] >> >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] >> >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] >> >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] >> >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] >> >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] >> >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] >> >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] >> >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] >> >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] >> >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] >> >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] >> >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] >> >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] >> >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] >> >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] >> >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] >> >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] >> >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] >> >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] >> >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] >> >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] >> >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] >> >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] >> >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] >> >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] >> >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] >> >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] >> >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] >> >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] >> >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] >> >> >> >> PrintGCStats output >> >> what count total mean max stddev >> >> gen0(s) 1282 776.553 0.60574 3.605 0.3765 >> >> gen1t(s) 823 1643.837 1.99737 11.139 1.0934 >> >> GC(s) 2105 2420.391 1.14983 11.139 1.0074 >> >> alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 >> >> promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 >> >> >> >> alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 MB/s >> >> alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 MB/s >> >> alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 MB/s >> >> promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 MB/s >> >> promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 MB/s >> >> gc_seq_load = 9681.563 s / 227507.228 s = 4.255% >> >> gc_conc_load = 0.000 s / 227507.228 s = 0.000% >> >> gc_tot_load = 9681.563 s / 227507.228 s = 4.255% >> >> >>> Thankx >>> >> >> >> /Kurt Shaffer/ >> UNIX Architect >> IT Enterprise Operations >> STERIS Corporation >> 440-392-7089 >> >> Internal Sites >> >> * UNIX Operations >> >> * STERIS Blogs >> * STERIS Wiki >> * STERIS Java Instant Messanger >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- ---------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS BUR02-311 | | e-mail: tony.printezis at sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | ---------------------------------------------------------------------- e-mail client: Thunderbird (Solaris) _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Wed Mar 5 11:48:44 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Wed, 05 Mar 2008 11:48:44 -0800 Subject: Tuning advice In-Reply-To: <47CEF697.30902@sun.com> References: <47CEF669.2010707@Sun.COM> <47CEF697.30902@sun.com> Message-ID: Also have you tried -server (instead of the client jvm you are using) to see if & by how much it improves performance of the application? (once you've eliminated the RMI-induced GC's as stated below). -- ramki ----- Original Message ----- From: Tony Printezis Date: Wednesday, March 5, 2008 11:41 am Subject: Re: Tuning advice To: Jon Masamitsu Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt" > Kurt, > > Do you use RMI? By default, RMI calls Full GC every 60 seconds for the > > correct operation of its distributed GC algorithm. The default > behavior > has been changed in 6.0 to be something less frequent (once an hour I > > believe...). > > Tony > > Jon Masamitsu wrote: > > Kurt, > > > > The full GC's in the log fragment that you sent are at > > very regular intervals so there might be something > > in the system that is invoking System.gc(). Full > > GC's normally happen when the heap is full and > > the logs show that the heap is not yet full when > > the full GC's happen. If the frequency of the full > > GC's is a problem, you can use -XX:+DisableExplicitGC > > to effectively turn off System.gc(). > > > > Other than that, it does not look like GC is a problem. The > > full GC's are all under 6 seconds and are about 60 seconds > > apart so I don't see how that can be causing the rendering > > of a page to take 40 seconds. Are there other parts of the > > gc log that show longer and/or more frequent GC pauses? > > > > Jon > > > > Shaffer, Kurt wrote: > > > >> Hello, > >> > >> > >> > >> We are looking for some tuning recommendations for the following > >> environment. I am not very familiar with Java so I am seeking any > help. > >> > >> > >> > >> > >>> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > >>> > >>> java version "1.4.2_13" > >>> > >>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06) > >>> > >>> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) > >>> > >>> >>> > >>> overridable="true" value="$(wt.java.cmd.quoted) -server -classpath > >>> > >>> $(wt.java.clas > >>> > >>> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient > >>> > >>> -DHTTPClient.disableKeepAlives\=true > >>> > >>> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 > >>> > >>> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC > >>> > >>> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails > >>> > >>> -Xloggc:$(wt.logs.di > >>> > >>> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> > >>> > >>> Application is PTC Windchill > >>> > >>> We have seen very slow response times from the users standpoint, > when > >>> > >>> they login it takes up to 15 seconds to render the display, when > they > >>> > >>> are navigating through different views in their browser it can > take 40 > >>> > >>> seconds or longer to display pages. > >>> > >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] > >> > >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] > >> > >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] > >> > >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] > >> > >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] > >> > >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] > >> > >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] > >> > >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] > >> > >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] > >> > >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] > >> > >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] > >> > >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] > >> > >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] > >> > >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] > >> > >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] > >> > >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] > >> > >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] > >> > >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] > >> > >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] > >> > >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] > >> > >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] > >> > >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] > >> > >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] > >> > >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] > >> > >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] > >> > >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] > >> > >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] > >> > >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] > >> > >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] > >> > >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] > >> > >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] > >> > >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] > >> > >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] > >> > >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] > >> > >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] > >> > >> > >> > >> PrintGCStats output > >> > >> what count total mean max stddev > >> > >> gen0(s) 1282 776.553 0.60574 3.605 0.3765 > >> > >> gen1t(s) 823 1643.837 1.99737 11.139 1.0934 > >> > >> GC(s) 2105 2420.391 1.14983 11.139 1.0074 > >> > >> alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 > >> > >> promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 > >> > >> > >> > >> alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 MB/s > >> > >> alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 MB/s > >> > >> alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 MB/s > >> > >> promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 MB/s > >> > >> promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 MB/s > >> > >> gc_seq_load = 9681.563 s / 227507.228 s = 4.255% > >> > >> gc_conc_load = 0.000 s / 227507.228 s = 0.000% > >> > >> gc_tot_load = 9681.563 s / 227507.228 s = 4.255% > >> > >> > >>> Thankx > >>> > >> > >> > >> /Kurt Shaffer/ > >> UNIX Architect > >> IT Enterprise Operations > >> STERIS Corporation < > >> 440-392-7089 > >> > >> Internal Sites > >> > >> * UNIX Operations > >> < > >> * STERIS Blogs < > >> * STERIS Wiki < > >> * STERIS Java Instant Messanger < > >> > >> > >> > >> > >> ------------------------------------------------------------------------ > >> > >> _______________________________________________ > >> hotspot-gc-use mailing list > >> hotspot-gc-use at openjdk.java.net > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > >> > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > -- > ---------------------------------------------------------------------- > | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | > | | MS BUR02-311 | > | e-mail: tony.printezis at sun.com | 35 Network Drive | > | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | > ---------------------------------------------------------------------- > e-mail client: Thunderbird (Solaris) > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Kurt_Shaffer at steris.com Wed Mar 5 12:14:23 2008 From: Kurt_Shaffer at steris.com (Shaffer, Kurt) Date: Wed, 5 Mar 2008 15:14:23 -0500 Subject: Tuning advice In-Reply-To: Message-ID: Thankx everyone for the quick responses. All of our 1.4.2 are showing client and our 1.5.0 are showing server, not sure how that happened. For those asking about RMI, I'm not sure what that is. The application is Windchill, I'll have to ask the application folks if they know. For those asking about stats, do you mean the PrintGCStats output at the end of the email? Thankx again, it is greatly appreciated. -----Original Message----- From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] Sent: Wednesday, March 05, 2008 14:49 To: Shaffer, Kurt Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Tuning advice Also have you tried -server (instead of the client jvm you are using) to see if & by how much it improves performance of the application? (once you've eliminated the RMI-induced GC's as stated below). -- ramki ----- Original Message ----- From: Tony Printezis Date: Wednesday, March 5, 2008 11:41 am Subject: Re: Tuning advice To: Jon Masamitsu Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt" > Kurt, > > Do you use RMI? By default, RMI calls Full GC every 60 seconds for the > > correct operation of its distributed GC algorithm. The default > behavior > has been changed in 6.0 to be something less frequent (once an hour I > > believe...). > > Tony > > Jon Masamitsu wrote: > > Kurt, > > > > The full GC's in the log fragment that you sent are at > > very regular intervals so there might be something > > in the system that is invoking System.gc(). Full > > GC's normally happen when the heap is full and > > the logs show that the heap is not yet full when > > the full GC's happen. If the frequency of the full > > GC's is a problem, you can use -XX:+DisableExplicitGC > > to effectively turn off System.gc(). > > > > Other than that, it does not look like GC is a problem. The > > full GC's are all under 6 seconds and are about 60 seconds > > apart so I don't see how that can be causing the rendering > > of a page to take 40 seconds. Are there other parts of the > > gc log that show longer and/or more frequent GC pauses? > > > > Jon > > > > Shaffer, Kurt wrote: > > > >> Hello, > >> > >> > >> > >> We are looking for some tuning recommendations for the following > >> environment. I am not very familiar with Java so I am seeking any > help. > >> > >> > >> > >> > >>> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > >>> > >>> java version "1.4.2_13" > >>> > >>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_13-b06) > >>> > >>> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) > >>> > >>> >>> > >>> overridable="true" value="$(wt.java.cmd.quoted) -server -classpath > >>> > >>> $(wt.java.clas > >>> > >>> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient > >>> > >>> -DHTTPClient.disableKeepAlives\=true > >>> > >>> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 > >>> > >>> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC > >>> > >>> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails > >>> > >>> -Xloggc:$(wt.logs.di > >>> > >>> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> > >>> > >>> Application is PTC Windchill > >>> > >>> We have seen very slow response times from the users standpoint, > when > >>> > >>> they login it takes up to 15 seconds to render the display, when > they > >>> > >>> are navigating through different views in their browser it can > take 40 > >>> > >>> seconds or longer to display pages. > >>> > >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] > >> > >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] > >> > >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] > >> > >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] > >> > >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] > >> > >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] > >> > >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] > >> > >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] > >> > >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] > >> > >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] > >> > >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] > >> > >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] > >> > >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] > >> > >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] > >> > >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] > >> > >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] > >> > >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] > >> > >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] > >> > >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] > >> > >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] > >> > >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] > >> > >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] > >> > >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] > >> > >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] > >> > >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] > >> > >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] > >> > >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] > >> > >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] > >> > >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] > >> > >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] > >> > >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] > >> > >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] > >> > >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] > >> > >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] > >> > >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] > >> > >> > >> > >> PrintGCStats output > >> > >> what count total mean max stddev > >> > >> gen0(s) 1282 776.553 0.60574 3.605 0.3765 > >> > >> gen1t(s) 823 1643.837 1.99737 11.139 1.0934 > >> > >> GC(s) 2105 2420.391 1.14983 11.139 1.0074 > >> > >> alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 > >> > >> promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 > >> > >> > >> > >> alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 MB/s > >> > >> alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 MB/s > >> > >> alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 MB/s > >> > >> promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 MB/s > >> > >> promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 MB/s > >> > >> gc_seq_load = 9681.563 s / 227507.228 s = 4.255% > >> > >> gc_conc_load = 0.000 s / 227507.228 s = 0.000% > >> > >> gc_tot_load = 9681.563 s / 227507.228 s = 4.255% > >> > >> > >>> Thankx > >>> > >> > >> > >> /Kurt Shaffer/ > >> UNIX Architect > >> IT Enterprise Operations > >> STERIS Corporation < > >> 440-392-7089 > >> > >> Internal Sites > >> > >> * UNIX Operations > >> < > >> * STERIS Blogs < > >> * STERIS Wiki < > >> * STERIS Java Instant Messanger < > >> > >> > >> > >> > >> ------------------------------------------------------------------------ > >> > >> _______________________________________________ > >> hotspot-gc-use mailing list > >> hotspot-gc-use at openjdk.java.net > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > >> > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > -- > ---------------------------------------------------------------------- > | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | > | | MS BUR02-311 | > | e-mail: tony.printezis at sun.com | 35 Network Drive | > | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | > ---------------------------------------------------------------------- > e-mail client: Thunderbird (Solaris) > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Wed Mar 5 23:17:02 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Wed, 05 Mar 2008 23:17:02 -0800 Subject: Tuning advice In-Reply-To: References: Message-ID: > All of our 1.4.2 are showing client and our 1.5.0 are showing server, > not sure how that happened. This is because of "GC ergonomics"; see http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html You should explicitly use -server to get the best performance everywhere (unless becnhmarking reveals -client to be superior in your case because of specific application requirements). > > For those asking about RMI, I'm not sure what that is. The application > is Windchill, I'll have to ask the application folks if they know. PTC/Windchill, I am told, uses RMI quite a bit. However, if the JVM's perform whole heap GC's on a fairly regular basis, you can either use -XX:+DisableExplicitGC or set the RMI DGC interval very high. Distributed garbage will continue to be collected in the course of normal whole heap GC's at the constituent JVM's (albeit depending on the RMI footprint perhaps at some increase heap requirements). See also the new (in 6.0) option -XX:+EplicitGCInvokesConcurrent (works with CMS collector); it'll allow you to do a whole heap GC mostly-concurrently and thus not pay a large stop-world pause penalty; it can thus be used to collect distributed garbage promptly via RMI DGC even if the set of JVM's would not otherwise do whole heap collections on a sufficiently frequent basis. > > For those asking about stats, do you mean the PrintGCStats output at the > end of the email? (Something appears to be wrong with the PrintGCStats summary you posted.) Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps. -- ramki Sun Developer Support: http://developers.sun.com/services > > Thankx again, it is greatly appreciated. > > > -----Original Message----- > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > Sent: Wednesday, March 05, 2008 14:49 > To: Shaffer, Kurt > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Tuning advice > > > Also have you tried -server (instead of the client jvm you are using) > to > see if & by how much it improves performance of the application? (once > you've eliminated the RMI-induced GC's as stated below). > > -- ramki > > ----- Original Message ----- > From: Tony Printezis > Date: Wednesday, March 5, 2008 11:41 am > Subject: Re: Tuning advice > To: Jon Masamitsu > Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt" > > > > > Kurt, > > > > Do you use RMI? By default, RMI calls Full GC every 60 seconds for the > > > > > correct operation of its distributed GC algorithm. The default > > behavior > > has been changed in 6.0 to be something less frequent (once an hour > I > > > > believe...). > > > > Tony > > > > Jon Masamitsu wrote: > > > Kurt, > > > > > > The full GC's in the log fragment that you sent are at > > > very regular intervals so there might be something > > > in the system that is invoking System.gc(). Full > > > GC's normally happen when the heap is full and > > > the logs show that the heap is not yet full when > > > the full GC's happen. If the frequency of the full > > > GC's is a problem, you can use -XX:+DisableExplicitGC > > > to effectively turn off System.gc(). > > > > > > Other than that, it does not look like GC is a problem. The > > > full GC's are all under 6 seconds and are about 60 seconds > > > apart so I don't see how that can be causing the rendering > > > of a page to take 40 seconds. Are there other parts of the > > > gc log that show longer and/or more frequent GC pauses? > > > > > > Jon > > > > > > Shaffer, Kurt wrote: > > > > > >> Hello, > > >> > > >> > > >> > > >> We are looking for some tuning recommendations for the following > > > >> environment. I am not very familiar with Java so I am seeking > any > > help. > > >> > > >> > > >> > > >> > > >>> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > > >>> > > >>> java version "1.4.2_13" > > >>> > > >>> Java(TM) 2 Runtime Environment, Standard Edition (build > 1.4.2_13-b06) > > >>> > > >>> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) > > >>> > > >>> > >>> > > >>> overridable="true" value="$(wt.java.cmd.quoted) -server > -classpath > > >>> > > >>> $(wt.java.clas > > >>> > > >>> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient > > >>> > > >>> -DHTTPClient.disableKeepAlives\=true > > >>> > > >>> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 > > >>> > > >>> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC > > >>> > > >>> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps > -XX:+PrintGCDetails > > >>> > > >>> -Xloggc:$(wt.logs.di > > >>> > > >>> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> > > >>> > > >>> Application is PTC Windchill > > >>> > > >>> We have seen very slow response times from the users > standpoint, > > when > > >>> > > >>> they login it takes up to 15 seconds to render the display, > when > > they > > >>> > > >>> are navigating through different views in their browser it can > > > take 40 > > >>> > > >>> seconds or longer to display pages. > > >>> > > >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] > > >> > > >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] > > >> > > >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] > > >> > > >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] > > >> > > >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] > > >> > > >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] > > >> > > >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] > > >> > > >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] > > >> > > >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] > > >> > > >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] > > >> > > >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] > > >> > > >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] > > >> > > >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] > > >> > > >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] > > >> > > >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] > > >> > > >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] > > >> > > >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] > > >> > > >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] > > >> > > >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] > > >> > > >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] > > >> > > >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] > > >> > > >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] > > >> > > >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] > > >> > > >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] > > >> > > >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] > > >> > > >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] > > >> > > >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] > > >> > > >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] > > >> > > >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] > > >> > > >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] > > >> > > >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] > > >> > > >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] > > >> > > >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] > > >> > > >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] > > >> > > >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] > > >> > > >> > > >> > > >> PrintGCStats output > > >> > > >> what count total mean max stddev > > >> > > >> gen0(s) 1282 776.553 0.60574 3.605 0.3765 > > >> > > >> gen1t(s) 823 1643.837 1.99737 11.139 1.0934 > > >> > > >> GC(s) 2105 2420.391 1.14983 11.139 1.0074 > > >> > > >> alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 > > >> > > >> promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 > > >> > > >> > > >> > > >> alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 > MB/s > > >> > > >> alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 > MB/s > > >> > > >> alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 > MB/s > > >> > > >> promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 > MB/s > > >> > > >> promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 > MB/s > > >> > > >> gc_seq_load = 9681.563 s / 227507.228 s = 4.255% > > >> > > >> gc_conc_load = 0.000 s / 227507.228 s = 0.000% > > >> > > >> gc_tot_load = 9681.563 s / 227507.228 s = 4.255% > > >> > > >> > > >>> Thankx > > >>> > > >> > > >> > > >> /Kurt Shaffer/ > > >> UNIX Architect > > >> IT Enterprise Operations > > >> STERIS Corporation < > > >> 440-392-7089 > > >> > > >> Internal Sites > > >> > > >> * UNIX Operations > > >> < > > >> * STERIS Blogs < > > >> * STERIS Wiki < > > >> * STERIS Java Instant Messanger < > > >> > > >> > > >> > > >> > > >> > ------------------------------------------------------------------------ > > >> > > >> _______________________________________________ > > >> hotspot-gc-use mailing list > > >> hotspot-gc-use at openjdk.java.net > > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > >> > > > _______________________________________________ > > > hotspot-gc-use mailing list > > > hotspot-gc-use at openjdk.java.net > > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > > > -- > > ---------------------------------------------------------------------- > > | Tony Printezis, Staff Engineer | Sun Microsystems Inc. > | > > | | MS BUR02-311 > | > > | e-mail: tony.printezis at sun.com | 35 Network Drive > | > > | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA > | > > ---------------------------------------------------------------------- > > e-mail client: Thunderbird (Solaris) > > > > > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From linuxhippy at gmail.com Thu Mar 6 03:29:14 2008 From: linuxhippy at gmail.com (Clemens Eisserer) Date: Thu, 6 Mar 2008 12:29:14 +0100 Subject: Tuning advice In-Reply-To: References: Message-ID: <194f62550803060329n1aa8a281m9efe07c0c1d8351e@mail.gmail.com> Hi, This is a bit off-topic ... > -XX:+DisableExplicitGC I think this switch is rather dangerous - some parts of the JDK rely on explicit garbage collection (e.g. Direct-*Buffers, instances pointing to non-heap memory cleaned up by finalizers). - the code which is quite guaranteed to work is then lowered to "may work, may fail", depending on the type of workload. A API which would allow better control over the type of GC would be quite helpful, as well as maybe a Hotspot-internal API which would help controllnig native resources allocated outside the VM. lg Clemens _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Thu Mar 6 06:14:46 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Thu, 06 Mar 2008 06:14:46 -0800 Subject: Tuning advice In-Reply-To: References: Message-ID: <47CFFC56.4040000@Sun.COM> Y Srinivas Ramakrishna wrote: > ... > > (Something appears to be wrong with the PrintGCStats summary you posted.) > Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps. > If the output is from 1.4.2 for the UseParallelGC collector, what we saw maybe all there is. Recall that PrintGCStats for UseParallelGC was fixed in 1.5. _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Thu Mar 6 08:41:33 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Thu, 06 Mar 2008 08:41:33 -0800 Subject: Tuning advice In-Reply-To: References: Message-ID: > Do you mean that the "-XX:+PrintGCDetails and -XX:+PrintGCTimeStamps" > is > run on the command line with the PrintGCStats command or set in the java > config for the JVM? The latter. But see Jon Masamitsu's follow-up response about the GC logs from +UseParallelGC in 1.4.2; which may make this somewhat moot in this case. -- ramki _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Kurt_Shaffer at steris.com Thu Mar 6 06:17:09 2008 From: Kurt_Shaffer at steris.com (Shaffer, Kurt) Date: Thu, 6 Mar 2008 09:17:09 -0500 Subject: Tuning advice In-Reply-To: Message-ID: Do you mean that the "-XX:+PrintGCDetails and -XX:+PrintGCTimeStamps" is run on the command line with the PrintGCStats command or set in the java config for the JVM? -----Original Message----- From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] Sent: Thursday, March 06, 2008 02:17 To: Shaffer, Kurt Cc: hotspot-gc-use at openjdk.java.net Subject: Re: RE: Tuning advice > All of our 1.4.2 are showing client and our 1.5.0 are showing server, > not sure how that happened. This is because of "GC ergonomics"; see http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html You should explicitly use -server to get the best performance everywhere (unless becnhmarking reveals -client to be superior in your case because of specific application requirements). > > For those asking about RMI, I'm not sure what that is. The application > is Windchill, I'll have to ask the application folks if they know. PTC/Windchill, I am told, uses RMI quite a bit. However, if the JVM's perform whole heap GC's on a fairly regular basis, you can either use -XX:+DisableExplicitGC or set the RMI DGC interval very high. Distributed garbage will continue to be collected in the course of normal whole heap GC's at the constituent JVM's (albeit depending on the RMI footprint perhaps at some increase heap requirements). See also the new (in 6.0) option -XX:+EplicitGCInvokesConcurrent (works with CMS collector); it'll allow you to do a whole heap GC mostly-concurrently and thus not pay a large stop-world pause penalty; it can thus be used to collect distributed garbage promptly via RMI DGC even if the set of JVM's would not otherwise do whole heap collections on a sufficiently frequent basis. > > For those asking about stats, do you mean the PrintGCStats output at the > end of the email? (Something appears to be wrong with the PrintGCStats summary you posted.) Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps. -- ramki Sun Developer Support: http://developers.sun.com/services > > Thankx again, it is greatly appreciated. > > > -----Original Message----- > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > Sent: Wednesday, March 05, 2008 14:49 > To: Shaffer, Kurt > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Tuning advice > > > Also have you tried -server (instead of the client jvm you are using) > to > see if & by how much it improves performance of the application? (once > you've eliminated the RMI-induced GC's as stated below). > > -- ramki > > ----- Original Message ----- > From: Tony Printezis > Date: Wednesday, March 5, 2008 11:41 am > Subject: Re: Tuning advice > To: Jon Masamitsu > Cc: hotspot-gc-use at openjdk.java.net, "Shaffer, Kurt" > > > > > Kurt, > > > > Do you use RMI? By default, RMI calls Full GC every 60 seconds for the > > > > > correct operation of its distributed GC algorithm. The default > > behavior > > has been changed in 6.0 to be something less frequent (once an hour > I > > > > believe...). > > > > Tony > > > > Jon Masamitsu wrote: > > > Kurt, > > > > > > The full GC's in the log fragment that you sent are at > > > very regular intervals so there might be something > > > in the system that is invoking System.gc(). Full > > > GC's normally happen when the heap is full and > > > the logs show that the heap is not yet full when > > > the full GC's happen. If the frequency of the full > > > GC's is a problem, you can use -XX:+DisableExplicitGC > > > to effectively turn off System.gc(). > > > > > > Other than that, it does not look like GC is a problem. The > > > full GC's are all under 6 seconds and are about 60 seconds > > > apart so I don't see how that can be causing the rendering > > > of a page to take 40 seconds. Are there other parts of the > > > gc log that show longer and/or more frequent GC pauses? > > > > > > Jon > > > > > > Shaffer, Kurt wrote: > > > > > >> Hello, > > >> > > >> > > >> > > >> We are looking for some tuning recommendations for the following > > > >> environment. I am not very familiar with Java so I am seeking > any > > help. > > >> > > >> > > >> > > >> > > >>> 4 cpus (Ultra Sparc III 900MHz) / 8GB memory > > >>> > > >>> java version "1.4.2_13" > > >>> > > >>> Java(TM) 2 Runtime Environment, Standard Edition (build > 1.4.2_13-b06) > > >>> > > >>> Java HotSpot(TM) Client VM (build 1.4.2_13-b06, mixed mode) > > >>> > > >>> > >>> > > >>> overridable="true" value="$(wt.java.cmd.quoted) -server > -classpath > > >>> > > >>> $(wt.java.clas > > >>> > > >>> spath.quoted) -Djava.protocol.handler.pkgs\=HTTPClient > > >>> > > >>> -DHTTPClient.disableKeepAlives\=true > > >>> > > >>> -DHTTPClient.dontChunkRequests\=true -Xss2M -Xms51 > > >>> > > >>> 2m -Xmx1024m -noverify -verbose:gc -XX:+UseParallelGC > > >>> > > >>> -XX:ParallelGCThreads=20 -XX:+PrintGCTimeStamps > -XX:+PrintGCDetails > > >>> > > >>> -Xloggc:$(wt.logs.di > > >>> > > >>> r)$(dir.sep)ms_gc.log wt.method.MethodServerMain"/> > > >>> > > >>> Application is PTC Windchill > > >>> > > >>> We have seen very slow response times from the users > standpoint, > > when > > >>> > > >>> they login it takes up to 15 seconds to render the display, > when > > they > > >>> > > >>> are navigating through different views in their browser it can > > > take 40 > > >>> > > >>> seconds or longer to display pages. > > >>> > > >> 63153.891: [GC 314184K->242537K(470016K), 0.5394510 secs] > > >> > > >> 63154.431: [Full GC 242537K->222908K(470464K), 4.3961010 secs] > > >> > > >> 63218.867: [GC 237602K->265484K(470464K), 0.4835425 secs] > > >> > > >> 63219.351: [Full GC 265484K->219722K(468800K), 4.0777246 secs] > > >> > > >> 63229.962: [GC 283466K->275130K(470016K), 0.9633742 secs] > > >> > > >> 63246.178: [GC 296442K->232743K(470016K), 0.2712703 secs] > > >> > > >> 63250.526: [GC 298919K->284345K(474432K), 0.6691910 secs] > > >> > > >> 63253.376: [GC 307833K->234235K(474432K), 0.7393747 secs] > > >> > > >> 63259.981: [GC 309243K->288777K(475584K), 0.7275010 secs] > > >> > > >> 63262.366: [GC 324297K->243675K(475584K), 0.5750282 secs] > > >> > > >> 63267.115: [GC 320987K->275409K(475712K), 1.0370391 secs] > > >> > > >> 63270.833: [GC 331025K->248161K(475712K), 0.8789762 secs] > > >> > > >> 63274.475: [GC 325729K->263489K(500032K), 1.6077160 secs] > > >> > > >> 63280.994: [GC 341057K->263713K(500032K), 0.9178316 secs] > > >> > > >> 63283.436: [GC 280071K->256830K(534464K), 0.6641916 secs] > > >> > > >> 63284.101: [Full GC 256830K->214330K(542528K), 4.6351180 secs] > > >> > > >> 63305.164: [GC 349882K->281490K(542528K), 0.6127869 secs] > > >> > > >> 63316.432: [GC 376658K->251450K(539136K), 1.0353315 secs] > > >> > > >> 63348.746: [GC 309133K->310170K(539136K), 0.3874349 secs] > > >> > > >> 63349.134: [Full GC 310170K->194499K(470208K), 4.9789538 secs] > > >> > > >> 63370.780: [GC 261059K->241509K(470784K), 0.4214626 secs] > > >> > > >> 63374.528: [GC 276453K->215817K(470784K), 0.8574863 secs] > > >> > > >> 63381.942: [GC 283529K->246533K(474816K), 0.6949210 secs] > > >> > > >> 63389.788: [GC 285893K->216402K(474816K), 1.3839849 secs] > > >> > > >> 63414.126: [GC 237126K->244639K(477696K), 0.6417762 secs] > > >> > > >> 63414.768: [Full GC 244639K->212594K(477696K), 4.6969125 secs] > > >> > > >> 63437.799: [GC 294130K->269346K(477760K), 0.5160500 secs] > > >> > > >> 63441.657: [GC 306082K->225578K(477760K), 0.3883250 secs] > > >> > > >> 63454.008: [GC 307242K->279893K(480256K), 0.4979514 secs] > > >> > > >> 63458.126: [GC 314261K->232665K(480256K), 0.5394576 secs] > > >> > > >> 63470.586: [GC 319321K->283449K(484096K), 0.6432362 secs] > > >> > > >> 63473.498: [GC 320953K->237395K(484096K), 0.2825034 secs] > > >> > > >> 63479.613: [GC 299104K->294563K(484096K), 0.5962983 secs] > > >> > > >> 63480.210: [Full GC 294563K->220268K(483584K), 5.8936777 secs] > > >> > > >> 63524.720: [GC 313580K->295967K(483904K), 0.7195508 secs] > > >> > > >> > > >> > > >> PrintGCStats output > > >> > > >> what count total mean max stddev > > >> > > >> gen0(s) 1282 776.553 0.60574 3.605 0.3765 > > >> > > >> gen1t(s) 823 1643.837 1.99737 11.139 1.0934 > > >> > > >> GC(s) 2105 2420.391 1.14983 11.139 1.0074 > > >> > > >> alloc(MB) 1282 -84202.223 -65.68036 169.752 101.4710 > > >> > > >> promo(MB) 1230 -86070.940 -69.97637 333.249 103.2167 > > >> > > >> > > >> > > >> alloc/elapsed_time = -84202.223 MB / 56876.807 s = -1.480 > MB/s > > >> > > >> alloc/tot_cpu_time = -84202.223 MB / 227507.228 s = -0.370 > MB/s > > >> > > >> alloc/mut_cpu_time = -84202.223 MB / 217825.665 s = -0.387 > MB/s > > >> > > >> promo/elapsed_time = -86070.940 MB / 56876.807 s = -1.513 > MB/s > > >> > > >> promo/gc0_time = -86070.940 MB / 0.000 s = 0.000 > MB/s > > >> > > >> gc_seq_load = 9681.563 s / 227507.228 s = 4.255% > > >> > > >> gc_conc_load = 0.000 s / 227507.228 s = 0.000% > > >> > > >> gc_tot_load = 9681.563 s / 227507.228 s = 4.255% > > >> > > >> > > >>> Thankx > > >>> > > >> > > >> > > >> /Kurt Shaffer/ > > >> UNIX Architect > > >> IT Enterprise Operations > > >> STERIS Corporation < > > >> 440-392-7089 > > >> > > >> Internal Sites > > >> > > >> * UNIX Operations > > >> < > > >> * STERIS Blogs < > > >> * STERIS Wiki < > > >> * STERIS Java Instant Messanger < > > >> > > >> > > >> > > >> > > >> > ------------------------------------------------------------------------ > > >> > > >> _______________________________________________ > > >> hotspot-gc-use mailing list > > >> hotspot-gc-use at openjdk.java.net > > >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > >> > > > _______________________________________________ > > > hotspot-gc-use mailing list > > > hotspot-gc-use at openjdk.java.net > > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > > > -- > > ---------------------------------------------------------------------- > > | Tony Printezis, Staff Engineer | Sun Microsystems Inc. > | > > | | MS BUR02-311 > | > > | e-mail: tony.printezis at sun.com | 35 Network Drive > | > > | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA > | > > ---------------------------------------------------------------------- > > e-mail client: Thunderbird (Solaris) > > > > > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Thu Mar 6 11:54:41 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Thu, 06 Mar 2008 11:54:41 -0800 Subject: Tuning advice In-Reply-To: <47CFFC56.4040000@Sun.COM> References: <47CFFC56.4040000@Sun.COM> Message-ID: <47D04C01.8090709@Sun.COM> Jon Masamitsu wrote: > > Y Srinivas Ramakrishna wrote: > ... >> (Something appears to be wrong with the PrintGCStats summary you posted.) >> Make sure you run with -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps. >> > > If the output is from 1.4.2 for the UseParallelGC collector, what we > saw maybe all there is. Recall that PrintGCStats for UseParallelGC This should have said "Recall that PrintGCDetails for ..." > was fixed in 1.5. > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Mon Mar 10 11:50:28 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Mon, 10 Mar 2008 11:50:28 -0700 Subject: Avoiding the Young Generation Guarantee Message-ID: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> All Our application scales very well when more memory is allocated to it. The Young Generation Guarantee limits that scalability as it forces Full GC while plenty of space is still available in the tenured space. That guarantee forces us to use relatively smaller nurseries. http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young %20Generation%20Guarantee%7Coutline infers that this guarantee only applies to the serial collector but I see it too with the parallel new collector. Maybe I am confused in the collector terminology. Could someone tell me how I can disable the young generation guarantee? Thanks in advance Alex A I have enclosed a snippet of the GC logs for analysis purposes. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20080310/25924472/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: YoungGenerationGuarantee-1.zip Type: application/x-zip-compressed Size: 813 bytes Desc: YoungGenerationGuarantee-1.zip Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20080310/25924472/attachment.bin -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Mon Mar 10 12:23:42 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Mon, 10 Mar 2008 12:23:42 -0700 Subject: Avoiding the Young Generation Guarantee In-Reply-To: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> Message-ID: -XX:+HandlePromotionFailure should allow you to circumvent the pessimal guarantee. (AFAIR, that was made the default behaviour in 6.0, but was not in 5.0, unless you also used the CMS collector; but the history is a bit vague in my mind, so you should of course check this rather than take my word for it.) -- ramki ----- Original Message ----- From: Alex Aisinzon Date: Monday, March 10, 2008 11:50 am Subject: Avoiding the Young Generation Guarantee To: hotspot-gc-use at openjdk.java.net > All > > > > Our application scales very well when more memory is allocated to it. > > The Young Generation Guarantee limits that scalability as it forces Full > GC while plenty of space is still available in the tenured space. > > That guarantee forces us to use relatively smaller nurseries. > > > http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young > %20Generation%20Guarantee%7Coutline infers that this guarantee only > applies to the serial collector but I see it too with the parallel new > collector. > > Maybe I am confused in the collector terminology. > > Could someone tell me how I can disable the young generation guarantee? > > > > Thanks in advance > > > > Alex A > > > > I have enclosed a snippet of the GC logs for analysis purposes. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Mon Mar 10 12:42:45 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Mon, 10 Mar 2008 12:42:45 -0700 Subject: Avoiding the Young Generation Guarantee In-Reply-To: <545E8962B7529546962672A564039F990C091840@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> <545E8962B7529546962672A564039F990C091840@exchange.guidewire.com> Message-ID: Relatedly, you might also want to (separately) try the +UseParallelGC collector -- in later versions of 5 (5u8 and later, AFAIR) it provides a parallel old collector as well: -XX:+UseParallelOldGC. Even otherwise, +UseParallelGC provides adaptive/dynamic sizing of the generations, which might provide you with better throughput than the static hand-tuning you would typically do with +UseParNewGC. The GC tuning guide provides further details (see "Ergonomics" and "Parallel Compaction"). -- ramki ----- Original Message ----- From: Alex Aisinzon Date: Monday, March 10, 2008 12:36 pm Subject: RE: Avoiding the Young Generation Guarantee To: Y Srinivas Ramakrishna Cc: hotspot-gc-use at openjdk.java.net > Srinivas > > I thank you for the pointer. I will try that option very soon and report > on it. > > Regards > > Alex Aisinzon > > -----Original Message----- > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > Sent: Monday, March 10, 2008 12:24 PM > To: Alex Aisinzon > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Avoiding the Young Generation Guarantee > > > -XX:+HandlePromotionFailure should allow you to circumvent the pessimal > guarantee. > (AFAIR, that was made the default behaviour in 6.0, but was not in 5.0, > unless > you also used the CMS collector; but the history is a bit vague in my > mind, > so you should of course check this rather than take my word for it.) > > -- ramki > > ----- Original Message ----- > From: Alex Aisinzon > Date: Monday, March 10, 2008 11:50 am > Subject: Avoiding the Young Generation Guarantee > To: hotspot-gc-use at openjdk.java.net > > > > All > > > > > > > > Our application scales very well when more memory is allocated to it. > > > > The Young Generation Guarantee limits that scalability as it forces > Full > > GC while plenty of space is still available in the tenured space. > > > > That guarantee forces us to use relatively smaller nurseries. > > > > > > > http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young > > %20Generation%20Guarantee%7Coutline infers that this guarantee only > > applies to the serial collector but I see it too with the parallel new > > collector. > > > > Maybe I am confused in the collector terminology. > > > > Could someone tell me how I can disable the young generation > guarantee? > > > > > > > > Thanks in advance > > > > > > > > Alex A > > > > > > > > I have enclosed a snippet of the GC logs for analysis purposes. > > > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Mon Mar 10 12:34:14 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Mon, 10 Mar 2008 12:34:14 -0700 Subject: Avoiding the Young Generation Guarantee In-Reply-To: <027FCB5D4C65CC4CA714042A4EE8CC6B03E5D41F@nzprm231.apac.corp.eds.com> References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> <027FCB5D4C65CC4CA714042A4EE8CC6B03E5D41F@nzprm231.apac.corp.eds.com> Message-ID: <545E8962B7529546962672A564039F990C091835@exchange.guidewire.com> Doug I thank you for your feedback. I use the ParNewGC, which according to Jon's blog (http://blogs.sun.com/jonthecollector/ ) is the combination of "ParNew" + "Serial Old". Therefore, I do not use the UseConcMarkSweepGC. I am using JDK 1.5.0_14. I would not expect the young generation to be applied. The logs seem to indicate that it is. Alex A ________________________________ From: Jones, Doug H [mailto:doug.jones at eds.com] Sent: Monday, March 10, 2008 12:15 PM To: Alex Aisinzon Subject: RE: Avoiding the Young Generation Guarantee Hi Alex, It is not supposed to be the case under JVM version 5.0 on, it definitely was under JVM 1.4.2. I'm assuming you are using ConcMarkSweepGC of the tenured area. Doug. ________________________________ From: hotspot-gc-use-bounces at openjdk.java.net [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Alex Aisinzon Sent: Tuesday, 11 March 2008 7:50 a.m. To: hotspot-gc-use at openjdk.java.net Subject: Avoiding the Young Generation Guarantee All Our application scales very well when more memory is allocated to it. The Young Generation Guarantee limits that scalability as it forces Full GC while plenty of space is still available in the tenured space. That guarantee forces us to use relatively smaller nurseries. http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young %20Generation%20Guarantee%7Coutline infers that this guarantee only applies to the serial collector but I see it too with the parallel new collector. Maybe I am confused in the collector terminology. Could someone tell me how I can disable the young generation guarantee? Thanks in advance Alex A I have enclosed a snippet of the GC logs for analysis purposes. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20080310/946961ad/attachment-0001.html -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Mon Mar 10 12:36:53 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Mon, 10 Mar 2008 12:36:53 -0700 Subject: Avoiding the Young Generation Guarantee In-Reply-To: References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> Message-ID: <545E8962B7529546962672A564039F990C091840@exchange.guidewire.com> Srinivas I thank you for the pointer. I will try that option very soon and report on it. Regards Alex Aisinzon -----Original Message----- From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] Sent: Monday, March 10, 2008 12:24 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Avoiding the Young Generation Guarantee -XX:+HandlePromotionFailure should allow you to circumvent the pessimal guarantee. (AFAIR, that was made the default behaviour in 6.0, but was not in 5.0, unless you also used the CMS collector; but the history is a bit vague in my mind, so you should of course check this rather than take my word for it.) -- ramki ----- Original Message ----- From: Alex Aisinzon Date: Monday, March 10, 2008 11:50 am Subject: Avoiding the Young Generation Guarantee To: hotspot-gc-use at openjdk.java.net > All > > > > Our application scales very well when more memory is allocated to it. > > The Young Generation Guarantee limits that scalability as it forces Full > GC while plenty of space is still available in the tenured space. > > That guarantee forces us to use relatively smaller nurseries. > > > http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young > %20Generation%20Guarantee%7Coutline infers that this guarantee only > applies to the serial collector but I see it too with the parallel new > collector. > > Maybe I am confused in the collector terminology. > > Could someone tell me how I can disable the young generation guarantee? > > > > Thanks in advance > > > > Alex A > > > > I have enclosed a snippet of the GC logs for analysis purposes. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Mon Mar 10 12:52:03 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Mon, 10 Mar 2008 12:52:03 -0700 Subject: Avoiding the Young Generation Guarantee In-Reply-To: References: <545E8962B7529546962672A564039F990C091743@exchange.guidewire.com> <545E8962B7529546962672A564039F990C091840@exchange.guidewire.com> Message-ID: <545E8962B7529546962672A564039F990C09189C@exchange.guidewire.com> Srinivas I thank you for the additional info. I was planning on testing UseParallelOldGC in our performance testing environment. I had noticed that this option was used in some Sun JDK SPECJBB2005 benchmarks (http://www.spec.org/jbb2005/results/res2006q1/jbb2005-20060117-00062.ht ml by example) The parallel nature of the young and tenured generation collection makes it a very good candidate for scaling across many processors and with large heaps, which is what I am looking for. Alex A -----Original Message----- From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] Sent: Monday, March 10, 2008 12:43 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: RE: Avoiding the Young Generation Guarantee Relatedly, you might also want to (separately) try the +UseParallelGC collector -- in later versions of 5 (5u8 and later, AFAIR) it provides a parallel old collector as well: -XX:+UseParallelOldGC. Even otherwise, +UseParallelGC provides adaptive/dynamic sizing of the generations, which might provide you with better throughput than the static hand-tuning you would typically do with +UseParNewGC. The GC tuning guide provides further details (see "Ergonomics" and "Parallel Compaction"). -- ramki ----- Original Message ----- From: Alex Aisinzon Date: Monday, March 10, 2008 12:36 pm Subject: RE: Avoiding the Young Generation Guarantee To: Y Srinivas Ramakrishna Cc: hotspot-gc-use at openjdk.java.net > Srinivas > > I thank you for the pointer. I will try that option very soon and report > on it. > > Regards > > Alex Aisinzon > > -----Original Message----- > From: Y Srinivas Ramakrishna [mailto:Y.S.Ramakrishna at Sun.COM] > Sent: Monday, March 10, 2008 12:24 PM > To: Alex Aisinzon > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Avoiding the Young Generation Guarantee > > > -XX:+HandlePromotionFailure should allow you to circumvent the pessimal > guarantee. > (AFAIR, that was made the default behaviour in 6.0, but was not in 5.0, > unless > you also used the CMS collector; but the history is a bit vague in my > mind, > so you should of course check this rather than take my word for it.) > > -- ramki > > ----- Original Message ----- > From: Alex Aisinzon > Date: Monday, March 10, 2008 11:50 am > Subject: Avoiding the Young Generation Guarantee > To: hotspot-gc-use at openjdk.java.net > > > > All > > > > > > > > Our application scales very well when more memory is allocated to it. > > > > The Young Generation Guarantee limits that scalability as it forces > Full > > GC while plenty of space is still available in the tenured space. > > > > That guarantee forces us to use relatively smaller nurseries. > > > > > > > http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#0.0.0.0.%20Young > > %20Generation%20Guarantee%7Coutline infers that this guarantee only > > applies to the serial collector but I see it too with the parallel new > > collector. > > > > Maybe I am confused in the collector terminology. > > > > Could someone tell me how I can disable the young generation > guarantee? > > > > > > > > Thanks in advance > > > > > > > > Alex A > > > > > > > > I have enclosed a snippet of the GC logs for analysis purposes. > > > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Kurt_Shaffer at steris.com Wed Mar 12 08:20:42 2008 From: Kurt_Shaffer at steris.com (Shaffer, Kurt) Date: Wed, 12 Mar 2008 11:20:42 -0400 Subject: Tuning advice In-Reply-To: <47D04C01.8090709@Sun.COM> Message-ID: Gurus, I really appreciate your advice, I've been asked a question that I can't answer. Even though no one has noticed a "problem area" from the data I've sent we are looking at trying some of the suggestions you all sent. The question is what order if any should they be attempted in? We have done two, setting the heap sizes larger and removed ParalletGCThreads so it defaults to 4 rather than the setting it used to be of 20 (4 cpu system). One of our Windchill apps person is saying that we are using RMI and there is not way of working around the fact it is forcing a full GC every 60 seconds except to upgrade to java 1.6.0. He also stated that suggestion 1 will only add another GC routine on top of the default full GC currently being done. I've attached an updated output from PrintGCStats at the bottom of the email. Recap of suggestions: 1) Set -XX:+UseConcMarkSweepGC to turn on the 'low pause' ConcurentMarkSweep Collector of the old area. 2) By default this will also turn on -XX:+UseParNewGC for the new area, so you will see lots of smaller ParNew Collections happening. This is OK, a desirable frequency is perhaps 2-3 secs. You should probably specify NewSize, the default with a 1GB heap will be 16mb which is normally too small. Say try 48mb, ie -XX:NewSize=48m. 3) I note you are using 1.4.2-13. I know there is a 'feature' in update 12 which causes CMS Collections to always be initiated when heap is half full. It is fixed in update 15, but I'm not sure about 13. This may not be a problem to you, except you are effectively just using approx half of the heap space. There is a workaround, specify -XX:CMSInitiatingOccupancyFraction=nn -XX:+UseCMSInitiatingOccupancyOnly=true. The default fraction is 68 (it is actually a percent). 4) Do you use RMI? By default, RMI calls Full GC every 60 seconds for the correct operation of its distributed GC algorithm. The default behavior has been changed in 6.0 to be something less frequent (once an hour I believe...). 5) Note that in 1.4.2_13 only the minor (young generation) collections are done on many processors; full collections are still serial collections. If your logs show those collections to be the problem, you might want to try the mostly-concurrent collector (-XX:+UseConcMarkSweepGC) instead, but that will require different tunings. PrintGCStats output (4 cpus / 3 JVMs / Windchill application): what count total mean max stddev gen0(s) 22 39.217 1.78260 2.710 0.4661 gen0t(s) 22 39.224 1.78289 2.711 0.4661 gen1t(s) 776 1845.592 2.37834 7.894 1.5820 GC(s) 798 1884.816 2.36192 7.894 1.5649 alloc(MB) 22 10289.001 467.68186 482.000 3.1980 promo(MB) 22 1427.789 64.89950 88.547 16.4298 alloc/elapsed_time = 10289.001 MB / 47197.783 s = 0.218 MB/s alloc/tot_cpu_time = 10289.001 MB / 188791.132 s = 0.054 MB/s alloc/mut_cpu_time = 10289.001 MB / 181251.870 s = 0.057 MB/s promo/elapsed_time = 1427.789 MB / 47197.783 s = 0.030 MB/s promo/gc0_time = 1427.789 MB / 39.224 s = 36.401 MB/s gc_seq_load = 7539.262 s / 188791.132 s = 3.993% gc_conc_load = 0.000 s / 188791.132 s = 0.000% gc_tot_load = 7539.262 s / 188791.132 s = 3.993% _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Wed Mar 12 10:41:18 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Wed, 12 Mar 2008 10:41:18 -0700 Subject: Tuning advice In-Reply-To: References: <47D04C01.8090709@Sun.COM> Message-ID: Hi Kurt -- > I really appreciate your advice, I've been asked a question that I can't > answer. Even though no one has noticed a "problem area" from the data > I've sent we are looking at trying some of the suggestions you all sent. The data indicates 22 scavenges @ 1.78 s each to 776 full collections @ 2.7 s each:- gen0t(s) 22 39.224 1.78289 2.711 0.4661 gen1t(s) 776 1845.592 2.37834 7.894 1.5820 Firstly, depending on your response-time needs you may want to tune the heap to reduce the scavenge pause times. Secondly, you are clearly spending way too much time 1846 seconds in full gc to just 39 seconds in scavenges. If, as was discussed before, the full gc's are all a result of RMI, the current settings for RMI are indeed very wasteful. Recall that distributed gc requires whole heap gc's to occur at each of several jvm's communicating via RMI with a certain promptness to allow the collection of distributed garbage. However, perhaps, as mentioned below, that frequency is too high by default. Try tuning it up via, for example :- -Dsun.rmi.dgc.server.gcInterval=600000 -Dsun.rmi.dgc.client.gcInterval=600000 You will have to tune it up or down depending on the heap occupancy and how well it works for your application configuration etc. As stated in:- http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6200091 the value was increased to:- Default values for sun.rmi.dgc.client.gcInterval sun.rmi.dgc.server.gcInterval are now 3600000 (one hour). Posted Date : 2005-07-28 22:11:37.0 as part of 6.0. > The question is what order if any should they be attempted in? We have > done two, setting the heap sizes larger and removed ParalletGCThreads > so > it defaults to 4 rather than the setting it used to be of 20 (4 cpu Setting parallel gc threads to 4 is a good first step (under the assumption that nothing else is running on the machine that can intefere). Setting the heap sizes larger will certainly reduce scavenge overheads; however, you may need to tune to meet your pause/responce time needs. > system). One of our Windchill apps person is saying that we are using > RMI and there is not way of working around the fact it is forcing a full > GC every 60 seconds except to upgrade to java 1.6.0. He also stated Please see the suggestion made above (and as already alluded to in previous discussion). > that suggestion 1 will only add another GC routine on top of the default > full GC currently being done. I've attached an updated output from That is true. Pre-6.0 (where -XX:+ExplicitGCInvokesConcurrent was introduced), just -XX:+UseConcMarkSweepGC would not work around the RMI-induced full gc's. HTHS. -- ramki > PrintGCStats at the bottom of the email. Recap of suggestions: > > 1) Set -XX:+UseConcMarkSweepGC to turn on the 'low pause' > ConcurentMarkSweep Collector of the old area. > > 2) By default this will also turn on -XX:+UseParNewGC for the new area, > so you will see lots of smaller ParNew Collections happening. This is > OK, a desirable frequency is perhaps 2-3 secs. You should probably > specify NewSize, the default with a 1GB heap will be 16mb which is > normally too small. Say try 48mb, ie -XX:NewSize=48m. > > 3) I note you are using 1.4.2-13. I know there is a 'feature' in update > 12 which causes CMS Collections to always be initiated when heap is half > full. It is fixed in update 15, but I'm not sure about 13. This may not > be a problem to you, except you are effectively just using approx half > of the heap space. There is a workaround, specify > -XX:CMSInitiatingOccupancyFraction=nn > -XX:+UseCMSInitiatingOccupancyOnly=true. > The default fraction is 68 (it is actually a percent). > > 4) Do you use RMI? By default, RMI calls Full GC every 60 seconds for > the > correct operation of its distributed GC algorithm. The default > behavior > has been changed in 6.0 to be something less frequent (once an hour I > > believe...). > > 5) Note that in 1.4.2_13 only the minor (young generation) collections > are done on many processors; full collections are still serial > collections. If your logs show those collections to be the problem, > you might want to try the mostly-concurrent collector > (-XX:+UseConcMarkSweepGC) instead, but that will require different > tunings. > > > > PrintGCStats output (4 cpus / 3 JVMs / Windchill application): > > what count total mean max stddev > gen0(s) 22 39.217 1.78260 2.710 0.4661 > gen0t(s) 22 39.224 1.78289 2.711 0.4661 > gen1t(s) 776 1845.592 2.37834 7.894 1.5820 > GC(s) 798 1884.816 2.36192 7.894 1.5649 > alloc(MB) 22 10289.001 467.68186 482.000 3.1980 > promo(MB) 22 1427.789 64.89950 88.547 16.4298 > > alloc/elapsed_time = 10289.001 MB / 47197.783 s = 0.218 MB/s > alloc/tot_cpu_time = 10289.001 MB / 188791.132 s = 0.054 MB/s > alloc/mut_cpu_time = 10289.001 MB / 181251.870 s = 0.057 MB/s > promo/elapsed_time = 1427.789 MB / 47197.783 s = 0.030 MB/s > promo/gc0_time = 1427.789 MB / 39.224 s = 36.401 MB/s > gc_seq_load = 7539.262 s / 188791.132 s = 3.993% > gc_conc_load = 0.000 s / 188791.132 s = 0.000% > gc_tot_load = 7539.262 s / 188791.132 s = 3.993% > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Keith.Holdaway at sas.com Wed Mar 12 10:32:58 2008 From: Keith.Holdaway at sas.com (Keith Holdaway) Date: Wed, 12 Mar 2008 13:32:58 -0400 Subject: Tuning advice In-Reply-To: References: <47D04C01.8090709@Sun.COM> Message-ID: <304E9E55F6A4BE4B910C2437D4D1B496090815BC07@MERCMBX14.na.sas.com> See comments embedded below: Keith R Holdaway Java Development Technologies SAS The Power to Know Carpe Diem -----Original Message----- From: hotspot-gc-dev-bounces at openjdk.java.net [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Shaffer, Kurt Sent: Wednesday, March 12, 2008 11:21 AM To: hotspot-gc-use at openjdk.java.net Subject: RE: Tuning advice Gurus, I really appreciate your advice, I've been asked a question that I can't answer. Even though no one has noticed a "problem area" from the data I've sent we are looking at trying some of the suggestions you all sent. The question is what order if any should they be attempted in? We have done two, setting the heap sizes larger and removed ParalletGCThreads so it defaults to 4 rather than the setting it used to be of 20 (4 cpu system). One of our Windchill apps person is saying that we are using RMI and there is not way of working around the fact it is forcing a full GC every 60 seconds except to upgrade to java 1.6.0. [Keith Holdaway] Ridiculous statement :) USE: -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 which enforces Distributed GC to ONE hour. He also stated that suggestion 1 will only add another GC routine on top of the default full GC currently being done. I've attached an updated output from PrintGCStats at the bottom of the email. Recap of suggestions: [Keith Holdaway] Don't listen to this guy, he obviously has no clue about GC! CMS is an Old Gen GC algorithm that if precluded from completing its several phases of work, will be interrupted by a FUll GC that collects Young and Old generations. CMS is used in the OG to reduce pauses seen sometimes by the default mark/sweep/compact algorithm in the OG. 1) Set -XX:+UseConcMarkSweepGC to turn on the 'low pause' ConcurentMarkSweep Collector of the old area. 2) By default this will also turn on -XX:+UseParNewGC for the new area, so you will see lots of smaller ParNew Collections happening. This is OK, a desirable frequency is perhaps 2-3 secs. You should probably specify NewSize, the default with a 1GB heap will be 16mb which is normally too small. Say try 48mb, ie -XX:NewSize=48m. 3) I note you are using 1.4.2-13. I know there is a 'feature' in update 12 which causes CMS Collections to always be initiated when heap is half full. It is fixed in update 15, but I'm not sure about 13. This may not be a problem to you, except you are effectively just using approx half of the heap space. There is a workaround, specify -XX:CMSInitiatingOccupancyFraction=nn -XX:+UseCMSInitiatingOccupancyOnly=true. The default fraction is 68 (it is actually a percent). 4) Do you use RMI? By default, RMI calls Full GC every 60 seconds for the correct operation of its distributed GC algorithm. The default behavior has been changed in 6.0 to be something less frequent (once an hour I believe...). 5) Note that in 1.4.2_13 only the minor (young generation) collections are done on many processors; full collections are still serial collections. If your logs show those collections to be the problem, you might want to try the mostly-concurrent collector (-XX:+UseConcMarkSweepGC) instead, but that will require different tunings. PrintGCStats output (4 cpus / 3 JVMs / Windchill application): what count total mean max stddev gen0(s) 22 39.217 1.78260 2.710 0.4661 gen0t(s) 22 39.224 1.78289 2.711 0.4661 gen1t(s) 776 1845.592 2.37834 7.894 1.5820 GC(s) 798 1884.816 2.36192 7.894 1.5649 alloc(MB) 22 10289.001 467.68186 482.000 3.1980 promo(MB) 22 1427.789 64.89950 88.547 16.4298 alloc/elapsed_time = 10289.001 MB / 47197.783 s = 0.218 MB/s alloc/tot_cpu_time = 10289.001 MB / 188791.132 s = 0.054 MB/s alloc/mut_cpu_time = 10289.001 MB / 181251.870 s = 0.057 MB/s promo/elapsed_time = 1427.789 MB / 47197.783 s = 0.030 MB/s promo/gc0_time = 1427.789 MB / 39.224 s = 36.401 MB/s gc_seq_load = 7539.262 s / 188791.132 s = 3.993% gc_conc_load = 0.000 s / 188791.132 s = 0.000% gc_tot_load = 7539.262 s / 188791.132 s = 3.993% _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From fw at deneb.enyo.de Wed Mar 12 23:58:14 2008 From: fw at deneb.enyo.de (Florian Weimer) Date: Thu, 13 Mar 2008 07:58:14 +0100 Subject: Tuning advice In-Reply-To: (Kurt Shaffer's message of "Wed, 12 Mar 2008 11:20:42 -0400") References: Message-ID: <87fxuv6rg9.fsf@mid.deneb.enyo.de> * Kurt Shaffer: > 1) Set -XX:+UseConcMarkSweepGC to turn on the 'low pause' > ConcurentMarkSweep Collector of the old area. This doesn't help on its own because you seem to be dealing with Full GCs and explicit GC invocations. The latter result in Full GCs even with the CMS collector by default. I think a switch to turn those explicit GC invocations into concurrent ones has recently been posted, but I can't find the message right now. _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Thu Mar 13 05:35:32 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Thu, 13 Mar 2008 05:35:32 -0700 Subject: Tuning advice In-Reply-To: <87fxuv6rg9.fsf@mid.deneb.enyo.de> References: <87fxuv6rg9.fsf@mid.deneb.enyo.de> Message-ID: <47D91F94.2040301@sun.com> Florian Weimer wrote On 03/12/08 23:58,: >* Kurt Shaffer: > > > >>1) Set -XX:+UseConcMarkSweepGC to turn on the 'low pause' >>ConcurentMarkSweep Collector of the old area. >> >> > >This doesn't help on its own because you seem to be dealing with Full >GCs and explicit GC invocations. The latter result in Full GCs even >with the CMS collector by default. > >I think a switch to turn those explicit GC invocations into concurrent >ones has recently been posted, but I can't find the message right now. > > In jdk6 you can turn on -XX:+ExplicitGCInvokesConcurrent with the CMS collector to do concurrent collections for System.gc() instead of the stop-the-world Full GC. _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From michael.finocchiaro at gmail.com Thu Mar 13 11:44:28 2008 From: michael.finocchiaro at gmail.com (Michael Finocchiaro) Date: Thu, 13 Mar 2008 13:44:28 -0500 Subject: Tuning advice In-Reply-To: <47D91F94.2040301@sun.com> References: <87fxuv6rg9.fsf@mid.deneb.enyo.de> <47D91F94.2040301@sun.com> Message-ID: <8b61e5430803131144m858f60eub7015684d37cce04@mail.gmail.com> If we are still talking about Windchill, no Java6 support until R9.1 this summer earliest so that option is irrelevant. Fino On 3/13/08, Jon Masamitsu wrote: > Florian Weimer wrote On 03/12/08 23:58,: > > >* Kurt Shaffer: > > > > > > > >>1) Set -XX:+UseConcMarkSweepGC to turn on the 'low pause' > >>ConcurentMarkSweep Collector of the old area. > >> > >> > > > >This doesn't help on its own because you seem to be dealing with Full > >GCs and explicit GC invocations. The latter result in Full GCs even > >with the CMS collector by default. > > > >I think a switch to turn those explicit GC invocations into concurrent > >ones has recently been posted, but I can't find the message right now. > > > > > > In jdk6 you can turn on -XX:+ExplicitGCInvokesConcurrent with the CMS > collector to do concurrent collections for System.gc() instead of the > stop-the-world Full GC. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- Sent from Google Mail for mobile | mobile.google.com Michael Finocchiaro michael.finocchiaro at gmail.com Mobile Telephone: +33 6 67 90 64 39 MSN: le_fino at hotmail.com _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Fri Mar 14 15:06:14 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Fri, 14 Mar 2008 15:06:14 -0700 Subject: Adaptiveness of adaptive sizing Message-ID: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com> Hi Our application has a lot of long lived objects (large session, cache,..). The size of the tenured space grows through the usage period (typically work hours) and then goes down as users log out. Typically, a JVM supporting 600 concurrent users could have 600-800MB of rock bottom long lived objects in the tenured. I am interested in exploring the adaptive sizing policy. Nevertheless, since the tenured grows during the day, I wonder if the adaptive sizing policy will adapt the tenured and nursery spaces dynamically. Finally, I wonder which metrics an adaptive sizing policy uses to find the "optimal" ratio between nursery and tenured. We are using the Sun JDK 1.5 for now. Thanks in advance Alex Aisinzon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20080314/7780fb5a/attachment.html -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Fri Mar 14 16:18:26 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Fri, 14 Mar 2008 16:18:26 -0700 Subject: Adaptiveness of adaptive sizing In-Reply-To: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com> Message-ID: <47DB07C2.70806@sun.com> Alex, The adaptive size policy does not currently move space between the tenured generation and the nursery. Rather the adaptive size policy grows and shrink each generation within their specified minimum and maximum sizes to try an achieve a user specified pause time goal and throughput goal. Given that fact, are you still interested in more details on how the adaptive size policy makes it's decisions? Jon Alex Aisinzon wrote On 03/14/08 15:06,: > Hi > > Our application has a lot of long lived objects (large session, > cache,..). > > The size of the tenured space grows through the usage period > (typically work hours) and then goes down as users log out. > > Typically, a JVM supporting 600 concurrent users could have 600-800MB > of rock bottom long lived objects in the tenured. > > I am interested in exploring the adaptive sizing policy. Nevertheless, > since the tenured grows during the day, I wonder if the adaptive > sizing policy will adapt the tenured and nursery spaces dynamically. > > Finally, I wonder which metrics an adaptive sizing policy uses to find > the ?optimal? ratio between nursery and tenured. > > We are using the Sun JDK 1.5 for now. > > Thanks in advance > > Alex Aisinzon > >------------------------------------------------------------------------ > >_______________________________________________ >hotspot-gc-use mailing list >hotspot-gc-use at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Fri Mar 14 16:36:18 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Fri, 14 Mar 2008 16:36:18 -0700 Subject: Adaptiveness of adaptive sizing In-Reply-To: <47DB07C2.70806@sun.com> References: <545E8962B7529546962672A564039F990C265AAC@exchange.guidewire.com> <47DB07C2.70806@sun.com> Message-ID: <545E8962B7529546962672A564039F990C265BD5@exchange.guidewire.com> Jon I thank you for your feedback. Our apps like memory and we tend to let them use as much as possible, even at the cost of some infrequent longer pauses. Therefore, I do not think that there is an interest in dynamically managing the size of the generations within their limits. In that light, I do not want to bother you by asking clarifications on techniques I will most likely not use. Regards Alex A -----Original Message----- From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] Sent: Friday, March 14, 2008 4:18 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Adaptiveness of adaptive sizing Alex, The adaptive size policy does not currently move space between the tenured generation and the nursery. Rather the adaptive size policy grows and shrink each generation within their specified minimum and maximum sizes to try an achieve a user specified pause time goal and throughput goal. Given that fact, are you still interested in more details on how the adaptive size policy makes it's decisions? Jon Alex Aisinzon wrote On 03/14/08 15:06,: > Hi > > Our application has a lot of long lived objects (large session, > cache,..). > > The size of the tenured space grows through the usage period > (typically work hours) and then goes down as users log out. > > Typically, a JVM supporting 600 concurrent users could have 600-800MB > of rock bottom long lived objects in the tenured. > > I am interested in exploring the adaptive sizing policy. Nevertheless, > since the tenured grows during the day, I wonder if the adaptive > sizing policy will adapt the tenured and nursery spaces dynamically. > > Finally, I wonder which metrics an adaptive sizing policy uses to find > the "optimal" ratio between nursery and tenured. > > We are using the Sun JDK 1.5 for now. > > Thanks in advance > > Alex Aisinzon > >----------------------------------------------------------------------- - > >_______________________________________________ >hotspot-gc-use mailing list >hotspot-gc-use at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From kalle at rosenbaum.se Tue Mar 18 07:23:29 2008 From: kalle at rosenbaum.se (Karl Rosenbaum) Date: Tue, 18 Mar 2008 15:23:29 +0100 Subject: Long pauses with parallel collector Message-ID: <47DFD061.8000802@rosenbaum.se> Hi We're experiencing long pauses (>60 seconds) using the parallel collector and the serial compacting old generation collector (the default choice on a server-class machine, right?). Unfortunately we have no logs from the gc, since this is a customer's installation and we can't play around with that. But I do have memory statistics along with response time statistics. Before the problems started we didn't really notice the collections but recently the collections took more than 60 seconds. This is measured using wget on a "ping"-servlet that returns some memory statistics. The RESPTIME is measured by timing the wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of long freeMem = Runtime.getRuntime().freeMemory(); long totalMem = Runtime.getRuntime().totalMemory(); long allocMem = Runtime.getRuntime().maxMemory(); This is an example of the statistics before the problems started: TIME FREE MEM MEM TOTAL MEM ALLOCATED RESPTIME 10:59:49 1499233752 12724273152 12724273152 0.20 10:59:59 1431576712 12724273152 12724273152 0.16 11:00:09 1392930432 12724273152 12724273152 0.14 11:00:19 1385974568 12724273152 12724273152 0.13 11:00:30 1365510896 12724273152 12724273152 0.13 11:00:40 1248830048 12724273152 12724273152 0.19 11:00:50 1164298568 12724273152 12724273152 0.13 11:01:00 1122874896 12724273152 12724273152 0.15 11:01:10 1085027216 12724273152 12724273152 0.13 11:01:20 952254960 12724273152 12724273152 0.14 11:01:31 385638352 12724273152 12724273152 0.30 11:01:41 3905940544 12717326336 12717326336 0.13 11:01:51 3286915952 12717326336 12717326336 0.34 11:02:01 1862466680 12717326336 12717326336 0.17 11:02:12 1214561176 12717326336 12717326336 0.13 11:02:22 937846336 12717326336 12717326336 0.13 11:02:32 830626208 12717326336 12717326336 0.13 11:02:42 591847776 12717326336 12717326336 0.13 11:02:52 486858808 12717326336 12717326336 0.15 11:03:02 405118840 12717326336 12717326336 0.12 11:03:12 379997776 12717326336 12717326336 0.17 11:03:23 4231164944 12723027968 12723027968 0.13 11:03:33 2869501088 12723027968 12723027968 0.14 11:03:43 2848014968 12723027968 12723027968 0.15 11:03:53 2789025152 12723027968 12723027968 0.13 11:04:03 2141846392 12723027968 12723027968 0.14 11:04:13 1936386592 12723027968 12723027968 0.13 11:04:24 1922345160 12723027968 12723027968 0.13 11:04:34 1907951992 12723027968 12723027968 0.14 11:04:44 1884581280 12723027968 12723027968 0.15 11:04:54 1842599720 12723027968 12723027968 0.14 11:05:04 1833357144 12723027968 12723027968 0.14 11:05:14 819218528 12723027968 12723027968 0.15 11:05:25 766194152 12723027968 12723027968 0.14 11:05:35 738061968 12723027968 12723027968 0.14 11:05:45 716541120 12723027968 12723027968 0.13 11:05:55 690620328 12723027968 12723027968 0.13 11:06:05 601896808 12723027968 12723027968 0.13 11:06:15 3986587288 12721192960 12721192960 0.19 11:06:26 3981820120 12721192960 12721192960 0.19 11:06:36 3908686896 12721192960 12721192960 0.13 11:06:46 3896673928 12721192960 12721192960 0.13 This is an example of the statistics after the problems started: TIME FREE MEM MEM TOTAL MEM ALLOCATED RESPTIME 15:26:13 132104472 12878741504 12878741504 0.11 15:26:23 89372112 12878741504 12878741504 0.21 15:26:33 61732520 12878741504 12878741504 0.12 15:26:43 15401280 12878741504 12878741504 12.14 15:27:05 4074793624 12878741504 12878741504 64.57 15:28:20 3378764376 12878741504 12878741504 0.12 15:28:30 2858626704 12878741504 12878741504 0.11 15:28:40 491173584 12878741504 12878741504 0.11 15:28:50 414802056 12878741504 12878741504 0.92 15:29:01 298133928 12878741504 12878741504 0.20 15:29:12 4164253720 12878741504 12878741504 64.36 15:30:26 3662076152 12878741504 12878741504 0.15 15:30:36 1823630328 12878741504 12878741504 0.11 15:30:46 1338510584 12878741504 12878741504 0.12 15:30:56 505714912 12878741504 12878741504 0.11 15:31:07 320903520 12878741504 12878741504 0.11 15:31:17 4179464600 12878741504 12878741504 71.62 15:32:38 3001104680 12878741504 12878741504 0.47 15:32:49 2593399336 12878741504 12878741504 0.13 15:32:59 2478360368 12878741504 12878741504 0.11 15:33:09 2391502968 12878741504 12878741504 0.11 15:33:19 785466304 12878741504 12878741504 0.13 15:33:29 4004574384 12878741504 12878741504 62.87 15:34:42 2643636392 12878741504 12878741504 0.11 15:34:52 2387674096 12878741504 12878741504 0.15 15:35:03 2226430872 12878741504 12878741504 0.11 15:35:13 1016694456 12878741504 12878741504 0.11 15:35:23 728910648 12878741504 12878741504 0.11 15:35:33 238095528 12878741504 12878741504 7.11 15:35:50 215109328 12878741504 12878741504 0.11 15:36:00 4025070120 12878741504 12878741504 69.14 15:37:19 2294767768 12878741504 12878741504 0.12 15:37:30 1897269848 12878741504 12878741504 0.11 15:37:40 1832375800 12878741504 12878741504 0.30 15:37:50 1741032976 12878741504 12878741504 0.11 15:38:00 1641480032 12878741504 12878741504 0.18 15:38:10 774869968 12878741504 12878741504 7.20 15:38:28 634489992 12878741504 12878741504 0.10 15:38:38 274370880 12878741504 12878741504 3.11 15:38:51 68896328 12878741504 12878741504 0.11 15:39:01 31211120 12878741504 12878741504 0.15 15:39:11 3997617032 12878741504 12878741504 60.77 As you can see, every collection results in a >60 seconds pause time. The server have been restarted once between the two log snippets and they are from different dates. The problem lasted indefinitely and all collections seem to take this long. We have now applied the CMS collector to "kill" these symptoms, but we still need to understand what was going on here. Do you have any suggestions on what the cause of this might be? The server is a Solaris 10 sparc machine with 16GB memory and 4 cpus. The java version is 1.5.0_11 These are the java parameters: root at fp # pargs 10502 10502: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx12288m -Dmx=12288 -Djava.awt.h argv[0]: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java argv[1]: -Xmx12288m argv[2]: -Dmx=12288 argv[3]: -Djava.awt.headless=true argv[4]: -Dfile.encoding=UTF-8 argv[5]: -Djava.library.path=/opt/FOCAfp/lib argv[6]: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager argv[7]: -Djava.endorsed.dirs=/opt/FOCAtomcat/common/endorsed argv[8]: -classpath argv[9]: :/opt/FOCAtomcat/bin/bootstrap.jar:/opt/FOCAtomcat/bin/commons-logging-api.jar argv[10]: -Dcatalina.base=/opt/FOCAtomcat argv[11]: -Dcatalina.home=/opt/FOCAtomcat argv[12]: -Djava.io.tmpdir=/opt/FOCAtomcat/temp argv[13]: org.apache.catalina.startup.Bootstrap argv[14]: start Best regards, Karl Rosenbaum _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Tue Mar 18 12:37:37 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Tue, 18 Mar 2008 12:37:37 -0700 Subject: Long pauses with parallel collector In-Reply-To: <47DFD061.8000802@rosenbaum.se> References: <47DFD061.8000802@rosenbaum.se> Message-ID: Hi Karl -- When the long response times are seen, the free memory jumps from about ~300 MB free to about ~4GB free. So it would seem as though the 60 s pauses are likely the serial compaction of the 12 GB heap. I would suggest either using -XX:+UseParalleOldGC (to compact the heap in parallel on the 4 cpus), or if that does not help enough (recall that 1.5.0_XX was the first version of parallel compaction and many performance enhancements were made to it subsequently in 6uXX which are not in 1.5.0_XX), then try -XX:+UseConcMarkSweepGC. Tuning docs etc can be found starting from: http://java.sun.com/javase/technologies/hotspot/index.jsp http://java.sun.com/javase/technologies/hotspot/gc/index.jsp GC logs using -XX:+PrintGCDetails will of course clarify matters more. There may be other flags with each of the parallel old and cms collectors that would provide more statistics as to where the time is going. HTHS. -- ramki ----- Original Message ----- From: Karl Rosenbaum Date: Tuesday, March 18, 2008 7:59 am Subject: Long pauses with parallel collector To: hotspot-gc-use at openjdk.java.net Cc: mikael.lonneberg at telelogic.com, Mattias Fredsberg > Hi > > We're experiencing long pauses (>60 seconds) using the parallel > collector and the serial compacting old generation collector (the > default choice on a server-class machine, right?). Unfortunately we > have > no logs from the gc, since this is a customer's installation and we > can't play around with that. But I do have memory statistics along > with > response time statistics. Before the problems started we didn't really > > notice the collections but recently the collections took more than 60 > > seconds. This is measured using wget on a "ping"-servlet that returns > > some memory statistics. The RESPTIME is measured by timing the > wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of > > long freeMem = Runtime.getRuntime().freeMemory(); > long totalMem = Runtime.getRuntime().totalMemory(); > long allocMem = Runtime.getRuntime().maxMemory(); > > > This is an example of the statistics before the problems started: > > TIME FREE MEM MEM TOTAL MEM ALLOCATED RESPTIME > 10:59:49 1499233752 12724273152 12724273152 0.20 > 10:59:59 1431576712 12724273152 12724273152 0.16 > 11:00:09 1392930432 12724273152 12724273152 0.14 > 11:00:19 1385974568 12724273152 12724273152 0.13 > 11:00:30 1365510896 12724273152 12724273152 0.13 > 11:00:40 1248830048 12724273152 12724273152 0.19 > 11:00:50 1164298568 12724273152 12724273152 0.13 > 11:01:00 1122874896 12724273152 12724273152 0.15 > 11:01:10 1085027216 12724273152 12724273152 0.13 > 11:01:20 952254960 12724273152 12724273152 0.14 > 11:01:31 385638352 12724273152 12724273152 0.30 > 11:01:41 3905940544 12717326336 12717326336 0.13 > 11:01:51 3286915952 12717326336 12717326336 0.34 > 11:02:01 1862466680 12717326336 12717326336 0.17 > 11:02:12 1214561176 12717326336 12717326336 0.13 > 11:02:22 937846336 12717326336 12717326336 0.13 > 11:02:32 830626208 12717326336 12717326336 0.13 > 11:02:42 591847776 12717326336 12717326336 0.13 > 11:02:52 486858808 12717326336 12717326336 0.15 > 11:03:02 405118840 12717326336 12717326336 0.12 > 11:03:12 379997776 12717326336 12717326336 0.17 > 11:03:23 4231164944 12723027968 12723027968 0.13 > 11:03:33 2869501088 12723027968 12723027968 0.14 > 11:03:43 2848014968 12723027968 12723027968 0.15 > 11:03:53 2789025152 12723027968 12723027968 0.13 > 11:04:03 2141846392 12723027968 12723027968 0.14 > 11:04:13 1936386592 12723027968 12723027968 0.13 > 11:04:24 1922345160 12723027968 12723027968 0.13 > 11:04:34 1907951992 12723027968 12723027968 0.14 > 11:04:44 1884581280 12723027968 12723027968 0.15 > 11:04:54 1842599720 12723027968 12723027968 0.14 > 11:05:04 1833357144 12723027968 12723027968 0.14 > 11:05:14 819218528 12723027968 12723027968 0.15 > 11:05:25 766194152 12723027968 12723027968 0.14 > 11:05:35 738061968 12723027968 12723027968 0.14 > 11:05:45 716541120 12723027968 12723027968 0.13 > 11:05:55 690620328 12723027968 12723027968 0.13 > 11:06:05 601896808 12723027968 12723027968 0.13 > 11:06:15 3986587288 12721192960 12721192960 0.19 > 11:06:26 3981820120 12721192960 12721192960 0.19 > 11:06:36 3908686896 12721192960 12721192960 0.13 > 11:06:46 3896673928 12721192960 12721192960 0.13 > > > This is an example of the statistics after the problems started: > > TIME FREE MEM MEM TOTAL MEM ALLOCATED RESPTIME > 15:26:13 132104472 12878741504 12878741504 0.11 > 15:26:23 89372112 12878741504 12878741504 0.21 > 15:26:33 61732520 12878741504 12878741504 0.12 > 15:26:43 15401280 12878741504 12878741504 12.14 > 15:27:05 4074793624 12878741504 12878741504 64.57 > 15:28:20 3378764376 12878741504 12878741504 0.12 > 15:28:30 2858626704 12878741504 12878741504 0.11 > 15:28:40 491173584 12878741504 12878741504 0.11 > 15:28:50 414802056 12878741504 12878741504 0.92 > 15:29:01 298133928 12878741504 12878741504 0.20 > 15:29:12 4164253720 12878741504 12878741504 64.36 > 15:30:26 3662076152 12878741504 12878741504 0.15 > 15:30:36 1823630328 12878741504 12878741504 0.11 > 15:30:46 1338510584 12878741504 12878741504 0.12 > 15:30:56 505714912 12878741504 12878741504 0.11 > 15:31:07 320903520 12878741504 12878741504 0.11 > 15:31:17 4179464600 12878741504 12878741504 71.62 > 15:32:38 3001104680 12878741504 12878741504 0.47 > 15:32:49 2593399336 12878741504 12878741504 0.13 > 15:32:59 2478360368 12878741504 12878741504 0.11 > 15:33:09 2391502968 12878741504 12878741504 0.11 > 15:33:19 785466304 12878741504 12878741504 0.13 > 15:33:29 4004574384 12878741504 12878741504 62.87 > 15:34:42 2643636392 12878741504 12878741504 0.11 > 15:34:52 2387674096 12878741504 12878741504 0.15 > 15:35:03 2226430872 12878741504 12878741504 0.11 > 15:35:13 1016694456 12878741504 12878741504 0.11 > 15:35:23 728910648 12878741504 12878741504 0.11 > 15:35:33 238095528 12878741504 12878741504 7.11 > 15:35:50 215109328 12878741504 12878741504 0.11 > 15:36:00 4025070120 12878741504 12878741504 69.14 > 15:37:19 2294767768 12878741504 12878741504 0.12 > 15:37:30 1897269848 12878741504 12878741504 0.11 > 15:37:40 1832375800 12878741504 12878741504 0.30 > 15:37:50 1741032976 12878741504 12878741504 0.11 > 15:38:00 1641480032 12878741504 12878741504 0.18 > 15:38:10 774869968 12878741504 12878741504 7.20 > 15:38:28 634489992 12878741504 12878741504 0.10 > 15:38:38 274370880 12878741504 12878741504 3.11 > 15:38:51 68896328 12878741504 12878741504 0.11 > 15:39:01 31211120 12878741504 12878741504 0.15 > 15:39:11 3997617032 12878741504 12878741504 60.77 > > As you can see, every collection results in a >60 seconds pause time. > > The server have been restarted once between the two log snippets and > they are from different dates. The problem lasted indefinitely and all > > collections seem to take this long. > > We have now applied the CMS collector to "kill" these symptoms, but we > > still need to understand what was going on here. > > Do you have any suggestions on what the cause of this might be? > > The server is a Solaris 10 sparc machine with 16GB memory and 4 cpus. > > The java version is 1.5.0_11 > > These are the java parameters: > root at fp # pargs 10502 > > 10502: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java -Xmx12288m > -Dmx=12288 -Djava.awt.h > argv[0]: /usr/jdk/instances/jdk1.5.0/bin/sparcv9/java > argv[1]: -Xmx12288m > argv[2]: -Dmx=12288 > argv[3]: -Djava.awt.headless=true > argv[4]: -Dfile.encoding=UTF-8 > argv[5]: -Djava.library.path=/opt/FOCAfp/lib > argv[6]: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager > argv[7]: -Djava.endorsed.dirs=/opt/FOCAtomcat/common/endorsed > argv[8]: -classpath > argv[9]: > :/opt/FOCAtomcat/bin/bootstrap.jar:/opt/FOCAtomcat/bin/commons-logging-api.jar > argv[10]: -Dcatalina.base=/opt/FOCAtomcat > argv[11]: -Dcatalina.home=/opt/FOCAtomcat > argv[12]: -Djava.io.tmpdir=/opt/FOCAtomcat/temp > argv[13]: org.apache.catalina.startup.Bootstrap > argv[14]: start > > > Best regards, > Karl Rosenbaum > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From michael.finocchiaro at gmail.com Tue Mar 18 13:25:24 2008 From: michael.finocchiaro at gmail.com (Michael Finocchiaro) Date: Tue, 18 Mar 2008 21:25:24 +0100 Subject: Long pauses with parallel collector In-Reply-To: References: <47DFD061.8000802@rosenbaum.se> Message-ID: <8b61e5430803181325n1ef89575wf3bf9bfb83ed9178@mail.gmail.com> You may want to set -XX:+PrintTenuringDistribution or attach with VisualGC or VisualVM to check whether the survivor space should be increased to avoid premature tenuring of objects into old. Similarly, perhaps -Xmn should be increased depending of the lifespans of your objects. Note that -XX:+ParallelOldGC uses implicitly -XX:+UseParallelGC (the throughput collector) whereas -XX:+UseConcMarkSweepGC uses -XX:+UseParNewGC (the low-pause collector) so the characteristics of the collections will most likely change a bit. I don't know whether -XX:+UseParallelGC automagically also turns on -XX:+UseAdaptiveSizing or not. Ramki? In any case, since we can't see the cause of the FullGC, perhaps -XX:+HandlePromotionFailure could help if this is due to a Young Guarantee GC rather than an OldFull or OldTooFull GC. Another hypothesis could be that you are running out of Perm space. Perhaps you need to set -XX:PermSize=64M which would be equal to -XX:MaxPermSize default size of 64M as increases of Perm also cause FullGC. Lastly, is there any possibility of using multiple smaller JVMs to avoid the super-long pauses in this 12GB one? Cheers, Fino On Tue, Mar 18, 2008 at 8:37 PM, Y Srinivas Ramakrishna < Y.S.Ramakrishna at sun.com> wrote: > > Hi Karl -- > > When the long response times are seen, the free memory jumps from about > ~300 MB > free to about ~4GB free. So it would seem as though the 60 s pauses are > likely the > serial compaction of the 12 GB heap. I would suggest either using > -XX:+UseParalleOldGC > (to compact the heap in parallel on the 4 cpus), or if that does not help > enough (recall that > 1.5.0_XX was the first version of parallel compaction and many performance > enhancements > were made to it subsequently in 6uXX which are not in 1.5.0_XX), then try > -XX:+UseConcMarkSweepGC. > > Tuning docs etc can be found starting from: > > http://java.sun.com/javase/technologies/hotspot/index.jsp > http://java.sun.com/javase/technologies/hotspot/gc/index.jsp > > GC logs using -XX:+PrintGCDetails will of course clarify matters more. > There may be other > flags with each of the parallel old and cms collectors that would provide > more statistics > as to where the time is going. > > HTHS. > -- ramki > > > ----- Original Message ----- > From: Karl Rosenbaum > Date: Tuesday, March 18, 2008 7:59 am > Subject: Long pauses with parallel collector > To: hotspot-gc-use at openjdk.java.net > Cc: mikael.lonneberg at telelogic.com, Mattias Fredsberg < > mattias.fredsberg at focalpoint.se> > > > > Hi > > > > We're experiencing long pauses (>60 seconds) using the parallel > > collector and the serial compacting old generation collector (the > > default choice on a server-class machine, right?). Unfortunately we > > have > > no logs from the gc, since this is a customer's installation and we > > can't play around with that. But I do have memory statistics along > > with > > response time statistics. Before the problems started we didn't really > > > > notice the collections but recently the collections took more than 60 > > > > seconds. This is measured using wget on a "ping"-servlet that returns > > > > some memory statistics. The RESPTIME is measured by timing the > > wget-command. FREE MEM, MEM TOTAL and MEM ALLOCATED are the results of > > > > long freeMem = Runtime.getRuntime().freeMemory(); > > long totalMem = Runtime.getRuntime().totalMemory(); > > long allocMem = Runtime.getRuntime().maxMemory(); > > > > > > This is an example of the statistics before the problems started: > > > > TIME FREE MEM MEM TOTAL MEM ALLOCATED RESPTIME > > 10:59:49 1499233752 12724273152 12724273152 0.20 > > 10:59:59 1431576712 12724273152 12724273152 0.16 > > 11:00:09 1392930432 12724273152 12724273152 0.14 > > 11:00:19 1385974568 12724273152 12724273152 0.13 > > 11:00:30 1365510896 12724273152 12724273152 0.13 > > 11:00:40 1248830048 12724273152 12