On Wed, Jul 7, 2010 at 5:26 PM, Y. S. Ramakrishna <span dir="ltr"><<a href="mailto:y.s.ramakrishna@oracle.com">y.s.ramakrishna@oracle.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
<br>
On 07/07/10 17:18, Todd Lipcon wrote:<br>
...<div class="im"><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Looking at the graph you attached, it appears that the low-water mark stabilizes at somewhere between 4.5G and 5G. The configuration I'm running is to allocate 40% of the heap to Memstore and 20% of the heap to the LRU cache. For an 8G heap, this is 4.8GB. So, for this application it's somewhat expected that, as it runs, it will accumulate more and more data until it reaches this threshold. The data is, of course, not *permanent*, but it's reasonably long-lived, so it makes sense to me that it should go into the old generation.<br>
</blockquote>
<br></div>
Ah, i see. In that case, i think you could try using a slightly larger old<br>
gen. If the old gen stabilizes at 4.2 GB, we should allow as much for slop.<br>
i.e. make the old gen 8.4 GB (or whatever is the measured stable<br>
old gen occupancy), then add to that the young gen size, and use<br>
that for the whole heap. I would be even more aggressive<br>
and grant more to the old gen -- as i said earlier perhaps<br>
double the old gen from its present size. If that doesn;t work<br>
we know that something is amiss in the way we are going at this.<br>
If it works, we can iterate downwards from a config that we know<br>
works, down to what may be considered an acceptable space overhead<br>
for GC.<div class="im"><br></div></blockquote><div><br></div><div>OK, I can try some tests with cache configured for only 40% heap usage. Should I run these tests with CMS or G1?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
If you like, I can tune down those percentages to 20/20 instead of 20/40, and I think we'll see the same pattern, just stabilized around 3.2GB. This will probably delay the full GCs, but still eventually hit them. It's also way lower than we can really go - customers won't like "throwing away" 60% of the allocated heap to GC!<br>
</blockquote>
<br></div>
I understand that sentiment. I want us to get to a state where we are able<br>
to completely avoid the creeping fragmentation, if possible. There are<br>
other ways to tune for this, but they are more labour-intensive and tricky,<br>
and I would not want to go into that lightly. You might want to contact<br>
your Java support for help with that.<div class="im"><br></div></blockquote><div><br></div><div>Yep, we've considered various solutions involving managing our own ref-counted slices of a single pre-allocated byte array - essentially writing our own slab allocator. In theory this should make all of the GCable objects constrained to a small number of sizes, and thus prevent fragmentation, but it's quite a project to undertake :)</div>
<div><br></div><div>Regarding Java support, as an open source project we have no such luxury. Projects like HBase and Hadoop, though, are pretty visible to users as "big Java apps", so getting them working well on the GC front does good things for Java adoption in the database/distributed systems community, I think.</div>
<div><br></div></div>-Todd<br><br clear="all"><br>-- <br>Todd Lipcon<br>Software Engineer, Cloudera<br>