Hi Ramki. Answers inline below:<br><br><div class="gmail_quote">On Thu, Oct 11, 2012 at 11:30 PM, Srinivas Ramakrishna <span dir="ltr"><<a href="mailto:ysr1729@gmail.com" target="_blank">ysr1729@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>Todd, good question :-)<br><br>@Jesper et al, do you know the answer to Todd's question? I agree that exposing all of these stats via suitable JMX/Mbean interfaces would be quite useful.... The other possibility would be to log in the manner of HP's gc logs (CSV format with suitable header), or jstat logs, so parsing cost would be minimal. Then higher level, general tools like Kafka could consume the log/event streams, apply suitable filters and inform/alert interested monitoring agents.<br>
<br></blockquote><div><br></div><div>Parsing CSV is one possibility, but somewhat painful, because you have all the usual issues with log rolling, compatibility between versions, etc. Certainly better than parsing the internal dump format that PrintFLSStatistics exposes at the moment, though :)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">@Todd & Saroj: Can you perhaps give some scenarios on how you might make use of information such as this (more concretely say CMS fragmentation at a specific JVM)? Would it be used only for "read-only" monitoring and alerting, or do you see this as part of an automated data-centric control system of sorts. The answer is kind of important, because something like the latter can be accomplished today via gc log parsing (however kludgey that might be) and something like Kafka/Zookeeper. On the other hand, I am not sure if the latency of that kind of thing would fit well into a more automated and fast-reacting data center control system or load-balancer where a more direct JMX/MBean like interface might work better. Or was your interest purely of the "development-debugging-performance-measurement" kind, rather than of production JVMs? Anyway, thinking out loud here...<br>
<br></blockquote><div><br></div><div>Just to give some context, one of the main products where I work is software which monitors large Hadoop clusters. Most of our daemons are low-heap, but a few, notably the HBase Region Server, can have large heaps and suffer from fragmentation. I wrote a few blog posts about this last year (starting here: <a href="http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/">http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/</a>)</div>
<div><br></div><div>So, if we can monitor fragmentation, I think there would be two useful things we could do:</div><div><br></div><div>1) If we notice that the heap is becoming really fragmented, we know a full GC is imminent. HBase has the capability to shift load between servers at runtime -- so we could simply ask the load balancer to move all load off the fragmented server, initiate a full GC manually, and then move the region back.</div>
<div><br></div><div>Less gracefully, we could have the server do a clean shutdown, which would be handled by our normal fault tolerance. This is actually better than a lengthy GC pause, because we can detect a clean shutdown immediately whereas the GC pause will take 30+ seconds before various heartbeats/sessions expire.</div>
<div><br></div><div>2) Our monitoring software already measures various JVM metrics and exposes them to operators (eg percentage of time spent in GC, heap usage after last GC, etc). If an operator suspects that GC is an issue, he or she can watch this metric or even set an alert. For some use cases, a fragmentation-induced STW GC is nearly catastrophic. An administrator should be able to quickly look at one of these metrics and tell whether the fragmentation is stable or if it's creeping towards an STW, in which case they need to re-evaluate GC tuning, live set size, etc.</div>
<div><br></div><div>Hope that helps with motivation for the feature.</div><div><br></div><div>-Todd</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br><div class="gmail_quote">On Thu, Oct 11, 2012 at 9:11 PM, Todd Lipcon <span dir="ltr"><<a href="mailto:todd@cloudera.com" target="_blank">todd@cloudera.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Ramki,<div><br></div><div>Do you know if there's any plan to offer the FLS statistics as a metric via JMX or some other interface in the future? It would be nice to be able to monitor fragmentation without having to actually log and parse the gc logs.</div>
<div><br></div><div>-Todd<div><div><br><br><div class="gmail_quote">On Thu, Oct 11, 2012 at 7:50 PM, Srinivas Ramakrishna <span dir="ltr"><<a href="mailto:ysr1729@gmail.com" target="_blank">ysr1729@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In the absence of fragmentation, one would normally expect the max chunk size of the CMS generation<br>to stabilize at some reasonable value, say after some 10's of CMS GC cycles. If it doesn't, you should try<br>
and use a larger heap, or otherwise reshape the heap to reduce promotion rates. In my experience,<br>
CMS seems to work best if its "duty cycle" is of the order of 1-2 %, i.e. there are 50 to 100 times more<br>scavenges during the interval that it's not running vs the interva during which it is running.<br>
<br>
Have Nagios grep the GC log file w/PrintFLSStatistics=2 for the string "Max Chunk Size:" and pick the<br>numeric component of every (4n+1)th match. The max chunk size will typically cycle within a small band,<br>
once it has stabilized, returning always to a high value following a CMS cycle's completion. If the upper envelope<br>of this keeps steadily declining over some 10's of CMS GC cycles, then you are probably seeing fragmentation<br>
that will eventually succumb to fragmentation.<br><br>You can probably calibrate a threshold for the upper envelope so that if it falls below that threshold you will<br>be alerted by Nagios that a closer look is in order.<br>
<br>At least something along those lines should work. The toughest part is designing your "filter" to detect the<br>fall in the upper envelope. You will probably want to plot the metric, then see what kind of filter will detect<br>
the condition.... Sorry this isn't much concrete help, but hopefully it gives you some ideas to work in<br>the right direction...<br><br>-- ramki<br><br><div class="gmail_quote"><div><div>On Thu, Oct 11, 2012 at 4:27 PM, roz dev <span dir="ltr"><<a href="mailto:rozdev29@gmail.com" target="_blank">rozdev29@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>Hi All<div><br></div><div>I am using Java 6u23, with CMS GC. I see that sometime Application gets paused for longer time because of excessive heap fragmentation.</div>
<div><br></div><div>I have enabled PrintFLSStatistics flag and following is the log</div>
<div><br></div><div><div><br></div><div>2012-10-09T15:38:44.724-0400: 52404.306: [GC Before GC:</div><div>Statistics for BinaryTreeDictionary:</div><div>------------------------------------</div><div>Total Free Space: -668151027</div>
<div>Max Chunk Size: 1976112973</div><div>Number of Blocks: 175445</div><div>Av. Block Size: 20672</div><div>Tree Height: 78</div><div>Before GC:</div><div>Statistics for BinaryTreeDictionary:</div><div>------------------------------------</div>
<div>Total Free Space: 10926</div><div>Max Chunk Size: 1660</div><div>Number of Blocks: 22</div><div>Av. Block Size: 496</div><div>Tree Height: 7</div></div><div><br></div><div><br></div><div>I would like to know from people about the way they track Heap Fragmentation and how do we alert for this situation?</div>
<div><br></div><div>We use Nagios and I am wondering if there is a way to parse these logs and know the max chunk size so that we can alert for it.</div><div><br></div><div>Any inputs are welcome.</div><div><br></div><div>
-Saroj</div><div><br></div><div><br></div><div><br></div>
<br></div></div>_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br></blockquote></div><br>
<br>_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>Todd Lipcon<br>Software Engineer, Cloudera<br>
</font></span></div>
</blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Todd Lipcon<br>Software Engineer, Cloudera<br>