<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">I didn’t know about it - I’m go to try it out on some of the nodes in a test cluster<div><br><div><div>On Jul 15, 2014, at 7:12 PM, Pas <<a href="mailto:pasthelod@gmail.com">pasthelod@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div><div><div>Hello,<br><br></div>I was wondering, how come no one (else) uses ParGCCardsPerStrideChunk on large heaps? It's shown to decrease (precious STW) time spent during minor collections by ParNew. ( <a href="http://blog.ragozin.info/2012/03/secret-hotspot-option-improving-gc.html">http://blog.ragozin.info/2012/03/secret-hotspot-option-improving-gc.html</a> and also we started to use it on 20+ GB heaps and it was helpful, though we did a more than one setting at a time, so I can't say that it was just this setting.)<br>
<br></div>Regards,<br></div>Pas<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jun 21, 2014 at 6:52 PM, graham sanderson <span dir="ltr"><<a href="mailto:graham@vast.com" target="_blank">graham@vast.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Note this works great for us too … given formatting in this email is a bit flaky, I’ll refer you to our numbers I posted in a Cassandra issue I opened to add these flags as defaults for ParNew/CMS (on the appropriate JVMs)<div>
<br></div><div><a href="https://issues.apache.org/jira/browse/CASSANDRA-7432" target="_blank">https://issues.apache.org/jira/browse/CASSANDRA-7432</a></div><div><div class="h5"><br><div><div>On Jun 14, 2014, at 7:05 PM, graham sanderson <<a href="mailto:graham@vast.com" target="_blank">graham@vast.com</a>> wrote:</div>
<br><blockquote type="cite"><div style="word-wrap:break-word">Thanks for the answer Gustav,<div><br></div><div>The fact that you have been running in production for months makes me confident enough to try this on at least one our nodes… (this is actually cassandra)</div>
<div><br></div><div>Current GC related options are at the bottom - these nodes have 256G of RAM, and they aren’t swapping, and we are certainly used to a pause within the first 10 seconds or so, but the nodes haven’t even joined the ring yet, so we don’t really care. yeah ms != mx is bad; we want one heap size and to stick with it.</div>
<div><br></div><div>I will gather data via -XX:+CMSEdenChunksRecordAlways, however I’d be interested if a developer has an answer as to when we expect potential chunk recording… Otherwise I’ll have to go dig into the code a bit deeper - my assumption was that this call would not be in the inlined allocation code, but I had thought that even allocation of a new TLAB was inlined by the compilers - perhaps not.</div>
<div><br></div><div>Current GC related settings - note we were running with a lower CMSInitiatingOccupancyFraction until recently - seems to have gotten changed back by accident, but that is kind of tangential.</div><div>
<br></div><div><div>-Xms24576M</div><div>-Xmx24576M</div><div>-Xmn8192M</div><div>-XX:+HeapDumpOnOutOfMemoryError</div><div>-XX:+UseParNewGC</div><div>-XX:+UseConcMarkSweepGC</div><div>-XX:+CMSParallelRemarkEnabled</div><div>
-XX:SurvivorRatio=8</div><div>-XX:MaxTenuringThreshold=1</div><div>-XX:CMSInitiatingOccupancyFraction=70</div><div>-XX:+UseCMSInitiatingOccupancyOnly</div><div>-XX:+UseTLAB</div><div>-XX:+UseCondCardMark</div><div>-XX:+PrintGCDetails</div>
<div>-XX:+PrintGCDateStamps</div><div>-XX:+PrintHeapAtGC</div><div>-XX:+PrintTenuringDistribution</div><div>-XX:+PrintGCApplicationStoppedTime</div><div>-XX:+PrintPromotionFailure</div><div>-XX:PrintFLSStatistics=1</div><div>
-Xloggc:/var/log/cassandra/gc.log</div><div>-XX:+UseGCLogFileRotation</div><div>-XX:NumberOfGCLogFiles=30</div><div>-XX:GCLogFileSize=20M</div><div>-XX:+PrintGCApplicationConcurrentTime</div><div><br></div></div><div>Thanks, Graham</div>
<div><br></div><div>P.S. Note tuning here is rather interesting since we use this cassandra cluster for lots of different data with very different usage patterns - sometimes we’ll suddenly dump 50G of data in over the course of a few minutes. Also cassandra doesn’t really mind a node being paused for a while due to GC, but things get a little more annoying if they pause at the same time… even though promotion failure can we worse for us (that is a separate issue), we’ve seen STW pauses up to about 6-8 seconds in re mark (presumably when things go horribly wrong and you only get one chunk). Basically I’m on a mission to minimize all pauses, since their effects can propagate (timeouts are very short in a lot of places)</div>
<div><br></div><div>I will report back with my findings</div><div><br></div><div><div><div>On Jun 14, 2014, at 6:29 PM, Gustav Åkesson <<a href="mailto:gustav.r.akesson@gmail.com" target="_blank">gustav.r.akesson@gmail.com</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr">Hi,<div><br></div><div>Even though I won't answer all your questions I'd like to share my experience with these settings (plus additional thoughts) even though I haven't yet have had the time to dig into details.</div>
<div><br></div><div>We've been using these flags for several months in production (yes, Java 7 even before latest update release) and we've seen a lot of improvements for CMS old gen STW. During execution occasional initial mark of 1.5s could occur, but using these settings combined CMS pauses are consistently around ~100ms (on high-end machine as yours, they are 20-30ms). We're using 1gb and 2gb heaps with roughly half/half old/new. Obviously, YMMV but this is at least the behavior of this particular application - we've had nothing but positive outcome from using these settings. Additionally, the pauses are rather deterministic.</div>
<div><br></div><div>Not sure what your heap size settings are, but what I've also observed is that setting Xms != Xmx could also cause occasional long initial mark when heap capacity is slightly increased. I had a discussion a while back ( <a href="http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-February/001795.html" target="_blank">http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2014-February/001795.html</a> ) regarding this, and this seems to be an issue with CMS.</div>
<div><br></div><div>Also, swapping/paging is another factor which could cause indeterministic / occasional long STW GCs. If you're on Linux, try swappiness=0 and see if pauses get more stable.</div><div><br></div><div>
<br></div><div>Best Regards,</div><div>Gustav Åkesson</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 13, 2014 at 6:48 AM, graham sanderson <span dir="ltr"><<a href="mailto:graham@vast.com" target="_blank">graham@vast.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="word-wrap:break-word"><div>I was investigating abortable preclean timeouts in our app (and associated long remark pause) so had a look at the old jdk6 code I had on my box, wondered about recording eden chunks during certain eden slow allocation paths (I wasn’t sure if TLAB allocation is just a CAS bump), and saw what looked perfect in the latest code, so was excited to install 1.7.0_60-b19</div>
<div><br></div><div>I wanted to ask what you consider the stability of these two options to be (I’m pretty sure at least the first one is new in this release)</div><div><br></div><div>I have just installed locally on my mac, and am aware of <a href="http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021809" target="_blank">http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021809</a> which I could reproduce, however I wasn’t able to reproduce it without -XX:-<span style="background-color:rgb(255,255,255);white-space:pre-wrap">UseCMSCompactAtFullCollection (is this your understanding too?)</span></div>
<div><br></div><div>We are running our application with 8 gig young generation (6.4g eden), on boxes with 32 cores… so parallelism is good for short pauses</div><div><br></div><div>we already have</div><div><br></div><div>
<div>-XX:+UseParNewGC </div><div>-XX:+UseConcMarkSweepGC</div><div>-XX:+CMSParallelRemarkEnabled</div></div><div><br></div><div>we have seen a few long(isn) initial marks, so </div><div><br></div><div>-XX:+CMSParallelInitialMarkEnabled sounds good</div>
<div><br></div><div>as for </div><div><br></div><div>-XX:+CMSEdenChunksRecordAlways</div><div><br></div><div>my question is: what constitutes a slow path such an eden chunk is potentially recorded… TLAB allocation, or more horrific things; basically (and I’ll test our app with -XX:+CMSPrintEdenSurvivorChunks) is it likely that I’ll actually get less samples using -XX:+CMSEdenChunksRecordAlways in a highly multithread app than I would with sampling, or put another way… what sort of app allocation patterns if any might avoid the slow path altogether and might leave me with just one chunk?</div>
<div><br></div><div>Thanks,</div><div><br></div><div>Graham</div><div><br></div><div>P.S. less relevant I think, but our old generation is 16g</div><div>P.P.S. I suspect the abortable preclean timeouts mostly happen after a burst of very high allocation rate followed by an almost complete lull… this is one of the patterns that can happen in our application</div>
</div><br>_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div></div></blockquote></div><br></div></div></div><br>_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div></body></html>