<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 3, 2020, 21:07 Laurence Cable <<a href="mailto:larry.cable@oracle.com" target="_blank" rel="noreferrer">larry.cable@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
On 9/3/20 9:03 AM, Volker Simonis wrote:<br>
> Hi,<br>
><br>
> I'd like to get your opinion on a POC I've done in order to speed up<br>
> heap dumps on Linux:<br>
><br>
> <a href="https://bugs.openjdk.java.net/browse/JDK-8252768" rel="noreferrer noreferrer noreferrer" target="_blank">https://bugs.openjdk.java.net/browse/JDK-8252768</a><br>
> <a href="http://cr.openjdk.java.net/~simonis/webrevs/2020/8252768/" rel="noreferrer noreferrer noreferrer" target="_blank">http://cr.openjdk.java.net/~simonis/webrevs/2020/8252768/</a><br>
><br>
> Currently, heap dumps can be taken by the SA tools from a frozen<br>
> process or core file or directly from a running process with jcmd,<br>
> jconsole & JMX, jmap, etc. If the heap of a running process is dumped,<br>
> this happens at a safepoint (see VM_HeapDumper). Because the time to<br>
> produce a heap dump is roughly proportional to the size and fill ratio<br>
> of the heap, this leads to safepoint times which can range from ~100ms<br>
> for a 100mb heap to ~1s for a 1gb heap up to 15s and more for a 8gb<br>
> heap (measured on my Core i7 laptop with SSD).<br>
><br>
> One possibility to decrease the safepoint time is to offload the<br>
> dumping work to an asynchronous process. On Linux (and probably any<br>
> other OS which supports fork()) this can be achieved by forking and<br>
> offloading the heap dumping to the child process. Forking still needs<br>
> to happen at a safepoint, but forking is considerably faster compared<br>
> to the dumping process itself. The fork performance is still<br>
> proportional to the size of the original Java process because although<br>
> fork won't copy any memory pages, the kernel still needs to duplicate<br>
> the page table entries of the process.<br>
</blockquote></div></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">curious what is the. behavior of the parent/target JVM process "after"<br>
it executes the fork() at the safepoint? i.e what does it do next?<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">It just continues its life. It will periodically try to reap the child process, but apart from that it will just run on.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> Linux uses a “copy-on-write†technique for the creation of a forked<br>
> child process. This means that right after creation, the child process<br>
> will have exactly the same memory image like its parent process. But<br>
> at the same time, the child process won’t use any additional physical<br>
> memory, as long as it doesn’t change (i.e. writes into) its memory.<br>
> Since heap dumping only reads the child process's memory and then<br>
> exits immediately, this technique can be applied even if the Java<br>
> process already uses almost the whole free physical memory.<br>
><br>
> The POC I've created (see<br>
> <a href="http://cr.openjdk.java.net/~simonis/webrevs/2020/8252768/" rel="noreferrer noreferrer noreferrer" target="_blank">http://cr.openjdk.java.net/~simonis/webrevs/2020/8252768/</a>) decreases<br>
> the aforementioned ~100ms, ~1s and 15s for a 100mb, 1gb and 8gb heap<br>
> to ~3ms, ~15ms and ~60ms on my laptop which I think is significant.<br>
> You can try it out by using the new "-async" or "-async=true" option<br>
> of the "GC.heap_dump" jcmd command.<br>
><br>
> Of course this change will require a CSR for the additional jcmd<br>
> GC.heap_dump "-async" option which I'll be happy to create if there's<br>
> any interest in this enhancement. Also, logging in the child process<br>
> might potentially interfere with logging in the parent VM and probably<br>
> will have to be removed in the final version, but I've left it in for<br>
> now to better illustrate what's happening. Finally, we can't output<br>
> the size of the created dump any more if we are using asynchronous<br>
> dumping but from my point of view that's not such a big problem. Apart<br>
> from that, the POC works surprisingly well :)<br>
><br>
> Please let me know what you think and if there's something I've overlooked?<br>
><br>
> Best regards,<br>
> Volker<br>
><br>
> PS: by the way, asynchronous dumping combines just fine with<br>
> compressed dumps. So you can easily use "GC.heap_dump -async=true<br>
> -gz=6"<br>
</blockquote></div></div></div>