<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Thanks for the responses. I agree with Jenny that the gc logs doesn't seem to indicate that the pause was from a GC event.
<br>
<br>
Profiling the application and trying to get some native stack traces/ core dumps.<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF741756"><font face="Tahoma" color="#000000" size="2"><b>From:</b> Martin Makundi [martin.makundi@koodaripalvelut.com]<br>
<b>Sent:</b> Monday, August 25, 2014 10:07 PM<br>
<b>To:</b> Yu Zhang<br>
<b>Cc:</b> Krishnamurthy, Kannan; hotspot-gc-use@openjdk.java.net; kndkannan@gmail.com<br>
<b>Subject:</b> Re: Unexplained long stop the world pauses during concurrent marking step in G1 Collector<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Try running command kill -3 <pid> to see what's going on with the application threads.
<div><br>
</div>
<div>**<br>
Martin</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-08-26 4:48 GMT+03:00 Yu Zhang <span dir="ltr"><<a href="mailto:yu.zhang@oracle.com" target="_blank">yu.zhang@oracle.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div bgcolor="#FFFFFF">
<div>Kannan,<br>
<br>
The concurrent marking is concurrent, meaning it runs concurrently with the application.
<br>
You may see the time between<span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none">
[GC concurrent-mark-start] and </span><br>
<span>[GC concurrent-mark-stop] very long, maybe even some young gc happened during this time period. This is because the marking threads can be suspended.
<br>
<br>
></span><span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none">*2014-08-07T13:42:39.598-0400:
92192.348: Application time: 9.0448670 seconds**<br>
2014-08-07T13:42:39.601-0400: 92192.351: Total time for which application threads were stopped: 0.0029740 seconds<br>
This means the application has run for </span><span>9.0448670 seconds. And only being stopped for
</span><br>
<span>0.0029740 seconds.<br>
<br>
>From this log, gc did not stop the application.<br>
<br>
></span><span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none">Linux
"top" shows single CPU at 100% and rest of the CPUs at 0% </span><span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none"><span>during
the pause</span>. <br>
Maybe something in the application or OS is running on 1 cpu and blocking other threads.<br>
</span>
<pre cols="72">Thanks,
Jenny</pre>
On 8/18/2014 1:34 PM, Krishnamurthy, Kannan wrote:<br>
</div>
<blockquote type="cite">
<div style="direction:ltr; font-family:Tahoma; color:#000000; font-size:10pt"><span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none">Greetings,
<br>
<br>
We are experiencing unexplained/unknown long pauses (8 seconds) during concurrent marking step of G1 collector.
<br>
<br>
2014-08-07T13:42:30.552-0400: 92183.303: [GC concurrent-root-region-scan-start]<br>
2014-08-07T13:42:30.555-0400: 92183.305: [GC concurrent-root-region-scan-end, 0.0025230 secs]<br>
**2014-08-07T13:42:30.555-0400: 92183.305: [GC concurrent-mark-start]**<br>
**2014-08-07T13:42:39.598-0400: 92192.348: Application time: 9.0448670 seconds**<br>
2014-08-07T13:42:39.601-0400: 92192.351: Total time for which application threads were stopped: 0.0029740 seconds<br>
2014-08-07T13:42:39.603-0400: 92192.353: [GC pause (G1 Evacuation Pause) (young) 92192.354: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 7980, predicted base time: 28.19 ms, remaining time: 71.81 ms, target pause time: 100.00 ms<br>
<br>
<br>
`2014-08-07T13:42:30.555-0400: 92183.305` is when the concurrent mark starts, approximately after 2 seconds of this step the application starts to pause. However the GC logs claims the application was not paused during this window.
<br>
Linux "top" shows single CPU at 100% and rest of the CPUs at 0% </span><span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none"><span style="color:rgb(0,0,0); font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:17.804800033569336px; text-align:left; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); display:inline!important; float:none">during
the pause</span>. <br>
<br>
Any help in understanding the root cause of this issue is appreciated. <br>
<br>
Our target JVMS:<br>
<br>
java version "1.7.0_04"<br>
Java(TM) SE Runtime Environment (build 1.7.0_04-b20)<br>
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)<br>
<br>
java version "1.8.0_11"<br>
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)<br>
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)<br>
<br>
<br>
Our JVM options : <br>
<br>
-Xms20G -Xmx20G -Xss10M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:MarkStackSize=16M<br>
-XX:+UnlockDiagnosticVMOptions -XX:+G1PrintRegionLivenessInfo -XX:+TraceGCTaskThread<br>
-XX:+G1SummarizeConcMark -XX:+G1SummarizeRSetStats -XX:+G1TraceConcRefinement<br>
-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=65 <br>
-XX:ParallelGCThreads=24 -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails <br>
-XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution
<br>
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime<br>
-Xloggc:/common/logs/ocean-partition-gc.log<br>
<br>
<br>
Thanks and regards, <br>
Kannan<br>
<br>
</span></div>
<br>
<fieldset target="_blank"></fieldset> <br>
<pre>_______________________________________________
hotspot-gc-use mailing list
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a>
</pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
hotspot-gc-use mailing list<br>
<a href="mailto:hotspot-gc-use@openjdk.java.net" target="_blank">hotspot-gc-use@openjdk.java.net</a><br>
<a href="http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use" target="_blank">http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>