From naasd at cboe.com  Mon Dec  3 08:57:30 2007
From: naasd at cboe.com (Naas, Dave)
Date: Mon, 3 Dec 2007 10:57:30 -0600
Subject: PrintGCStats
Message-ID: <4BD174404CF4A34C98322DC926CF862B2289F2@MSMAIL.cboent.cboe.com>

Regarding the posting from Oct 26, 2007:  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/000064.html	
where Mr. Kessler states: 
quote 
	> > I'll see if we can get an updated version of PrintGCStats out to replace the ancient one available at 
	> >	 <http://java.sun.com/developer/technicalArticles/Programming/turbo/> 
	> > That should give you the kinds of statistics you are asking for. 
	> > 	... peter 
unquote

I was wondering if a new version of PrintGCStats and PrintGCFixup will indeed be made available or if we should proceed with our own updates to handle new jdk releases, etc.??    

A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics flag.  We have found that this setting prevents the current PrintGCStats from properly parsing ParNew data and thus doesn't generate the Promotion stats (Promo.csv file).   The following snippet shows how the ParNew line is now split (shown in bold), thanks to the -XX:PrintFLSStatistics:  

======================================================================================================================================================
{Heap before GC invocations=0 (full 0):
 par new generation   total 305088K, used 277376K [0x2f800000, 0x43d00000, 0x43d00000)
  eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
  from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
  to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
 concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000, 0xeb000000, 0xeb000000)
 concurrent-mark-sweep perm gen total 131072K, used 14913K [0xeb000000, 0xf3000000, 0xfb000000)
8.746: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 701235200
Max   Chunk Size: 701235200
Number of Blocks: 1
Av.  Block  Size: 701235200
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
8.746: [ParNew
Desired survivor size 14188544 bytes, new threshold 1 (max 4)
- age   1:   15763640 bytes,   15763640 total
- age   2:         56 bytes,   15763696 total
: 277376K->15474K(305088K), 0.5657873 secs] 277376K->15474K(3044288K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 701218816
Max   Chunk Size: 701218816
Number of Blocks: 1
Av.  Block  Size: 701218816
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
======================================================================================================================================================

Thank you, 
Dave 

 David Naas
Chicago Board Options Exchange 
 312-786-7222


From Peter.Kessler at Sun.COM  Mon Dec  3 10:19:41 2007
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Mon, 03 Dec 2007 10:19:41 -0800
Subject: PrintGCStats
In-Reply-To: <4BD174404CF4A34C98322DC926CF862B2289F2@MSMAIL.cboent.cboe.com>
References: <4BD174404CF4A34C98322DC926CF862B2289F2@MSMAIL.cboent.cboe.com>
Message-ID: <475448BD.3030401@Sun.COM>

It looks like the most recent PrintGCFixup (and PrintGCStats
and CompareGCStats) is "posted" in

     http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/51

You have to join all the lines that got wrapped (sigh).
Sometimes you have to run PrintGCFixup more than once to
get it to remove things so that it can see other things
to remove.  Though, in your case, running it twice doesn't
help remove the output of PrintFLSStatistics.  It does look
like if you add a cleaner for PrintFLSStatistics, you'll
have to run PrintGCFixup at least twice, since that output
gets in the way of the removal of the PrintTenuringDistribution
flag.

PrintGCFixup and friends look like a prime candidate for putting
in hotspot/tools/ and letting people make improvements.  If you
wanted to contribute your addition for removing PrintFLSStatistics,
I'm sure we could find someone here to sponsor it.

			... peter

Naas, Dave wrote:

> Regarding the posting from Oct 26, 2007:  http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/000064.html	
> where Mr. Kessler states: 
> quote 
> 	> > I'll see if we can get an updated version of PrintGCStats out to replace the ancient one available at 
> 	> >	 <http://java.sun.com/developer/technicalArticles/Programming/turbo/> 
> 	> > That should give you the kinds of statistics you are asking for. 
> 	> > 	... peter 
> unquote
> 
> I was wondering if a new version of PrintGCStats and PrintGCFixup will indeed be made available or if we should proceed with our own updates to handle new jdk releases, etc.??    
> 
> A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics flag.  We have found that this setting prevents the current PrintGCStats from properly parsing ParNew data and thus doesn't generate the Promotion stats (Promo.csv file).   The following snippet shows how the ParNew line is now split (shown in bold), thanks to the -XX:PrintFLSStatistics:  
> 
> ======================================================================================================================================================
> {Heap before GC invocations=0 (full 0):
>  par new generation   total 305088K, used 277376K [0x2f800000, 0x43d00000, 0x43d00000)
>   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
>   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
>   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
>  concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000, 0xeb000000, 0xeb000000)
>  concurrent-mark-sweep perm gen total 131072K, used 14913K [0xeb000000, 0xf3000000, 0xfb000000)
> 8.746: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701235200
> Max   Chunk Size: 701235200
> Number of Blocks: 1
> Av.  Block  Size: 701235200
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 8.746: [ParNew
> Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> - age   1:   15763640 bytes,   15763640 total
> - age   2:         56 bytes,   15763696 total
> : 277376K->15474K(305088K), 0.5657873 secs] 277376K->15474K(3044288K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701218816
> Max   Chunk Size: 701218816
> Number of Blocks: 1
> Av.  Block  Size: 701218816
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
> ======================================================================================================================================================
> 
> Thank you, 
> Dave 
> 
>  David Naas
> Chicago Board Options Exchange 
>  312-786-7222
> 



From Ciciora at cboe.com  Mon Dec  3 12:25:33 2007
From: Ciciora at cboe.com (Ciciora, Paul)
Date: Mon, 3 Dec 2007 14:25:33 -0600
Subject: PrintGCStats
References: <mailman.1.1196712000.22753.hotspot-gc-dev@openjdk.java.net>
Message-ID: <8CDFE9CE1B9A344E9AA47C27B7C58DD8168DD7@MSMAIL.cboent.cboe.com>

Since GC formats are constantly changing and different collectors log
stats differently I'd love to so you adopt a harvestable output format.
The idea would be to send stats to a separate file and use a name/value
style format. That way you could send whatever you wanted there and all
the harvester script or program would have to know is the tag name. The
name could be terse to save space. You could register the tag names so
open source changes would conform. If you did that then format changes
to the human readable output could be separate. You wouldn't have to be
as sensitive to breaking scripts.


Just a thought. I'm probably not the first one to suggest this.

-----Original Message-----
From: hotspot-gc-dev-bounces at openjdk.java.net
[mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of
hotspot-gc-dev-request at openjdk.java.net
Sent: Monday, December 03, 2007 2:00 PM
To: hotspot-gc-dev at openjdk.java.net
Subject: hotspot-gc-dev Digest, Vol 6, Issue 1

Send hotspot-gc-dev mailing list submissions to
	hotspot-gc-dev at openjdk.java.net

To subscribe or unsubscribe via the World Wide Web, visit
	http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev
or, via email, send a message with subject or body 'help' to
	hotspot-gc-dev-request at openjdk.java.net

You can reach the person managing the list at
	hotspot-gc-dev-owner at openjdk.java.net

When replying, please edit your Subject line so it is more specific
than "Re: Contents of hotspot-gc-dev digest..."


Today's Topics:

   1. PrintGCStats (Naas, Dave)
   2. Re: PrintGCStats (Peter B. Kessler)


----------------------------------------------------------------------

Message: 1
Date: Mon, 3 Dec 2007 10:57:30 -0600
From: "Naas, Dave" <naasd at cboe.com>
Subject: PrintGCStats
To: <hotspot-gc-dev at openjdk.java.net>
Message-ID:
	<4BD174404CF4A34C98322DC926CF862B2289F2 at MSMAIL.cboent.cboe.com>
Content-Type: text/plain;	charset="iso-8859-1"

Regarding the posting from Oct 26, 2007:
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
4.html	
where Mr. Kessler states: 
quote 
	> > I'll see if we can get an updated version of PrintGCStats
out to replace the ancient one available at 
	> >
<http://java.sun.com/developer/technicalArticles/Programming/turbo/> 
	> > That should give you the kinds of statistics you are asking
for. 
	> > 	... peter 
unquote

I was wondering if a new version of PrintGCStats and PrintGCFixup will
indeed be made available or if we should proceed with our own updates to
handle new jdk releases, etc.??    

A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics
flag.  We have found that this setting prevents the current PrintGCStats
from properly parsing ParNew data and thus doesn't generate the
Promotion stats (Promo.csv file).   The following snippet shows how the
ParNew line is now split (shown in bold), thanks to the
-XX:PrintFLSStatistics:  

========================================================================
========================================================================
======
{Heap before GC invocations=0 (full 0):
 par new generation   total 305088K, used 277376K [0x2f800000,
0x43d00000, 0x43d00000)
  eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
  from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
  to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
 concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000,
0xeb000000, 0xeb000000)
 concurrent-mark-sweep perm gen total 131072K, used 14913K [0xeb000000,
0xf3000000, 0xfb000000)
8.746: [GC Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 701235200
Max   Chunk Size: 701235200
Number of Blocks: 1
Av.  Block  Size: 701235200
Tree      Height: 1
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
8.746: [ParNew
Desired survivor size 14188544 bytes, new threshold 1 (max 4)
- age   1:   15763640 bytes,   15763640 total
- age   2:         56 bytes,   15763696 total
: 277376K->15474K(305088K), 0.5657873 secs]
277376K->15474K(3044288K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 701218816
Max   Chunk Size: 701218816
Number of Blocks: 1
Av.  Block  Size: 701218816
Tree      Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max   Chunk Size: 0
Number of Blocks: 0
Tree      Height: 0
, 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
========================================================================
========================================================================
======

Thank you, 
Dave 

 David Naas
Chicago Board Options Exchange 
 312-786-7222



------------------------------

Message: 2
Date: Mon, 03 Dec 2007 10:19:41 -0800
From: "Peter B. Kessler" <Peter.Kessler at Sun.COM>
Subject: Re: PrintGCStats
To: "Naas, Dave" <naasd at cboe.com>
Cc: hotspot-gc-dev at openjdk.java.net
Message-ID: <475448BD.3030401 at Sun.COM>
Content-Type: text/plain; format=flowed; charset=ISO-8859-1

It looks like the most recent PrintGCFixup (and PrintGCStats
and CompareGCStats) is "posted" in

 
http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/51

You have to join all the lines that got wrapped (sigh).
Sometimes you have to run PrintGCFixup more than once to
get it to remove things so that it can see other things
to remove.  Though, in your case, running it twice doesn't
help remove the output of PrintFLSStatistics.  It does look
like if you add a cleaner for PrintFLSStatistics, you'll
have to run PrintGCFixup at least twice, since that output
gets in the way of the removal of the PrintTenuringDistribution
flag.

PrintGCFixup and friends look like a prime candidate for putting
in hotspot/tools/ and letting people make improvements.  If you
wanted to contribute your addition for removing PrintFLSStatistics,
I'm sure we could find someone here to sponsor it.

			... peter

Naas, Dave wrote:

> Regarding the posting from Oct 26, 2007:
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
4.html	
> where Mr. Kessler states: 
> quote 
> 	> > I'll see if we can get an updated version of PrintGCStats
out to replace the ancient one available at 
> 	> >
<http://java.sun.com/developer/technicalArticles/Programming/turbo/> 
> 	> > That should give you the kinds of statistics you are asking
for. 
> 	> > 	... peter 
> unquote
> 
> I was wondering if a new version of PrintGCStats and PrintGCFixup will
indeed be made available or if we should proceed with our own updates to
handle new jdk releases, etc.??    
> 
> A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics
flag.  We have found that this setting prevents the current PrintGCStats
from properly parsing ParNew data and thus doesn't generate the
Promotion stats (Promo.csv file).   The following snippet shows how the
ParNew line is now split (shown in bold), thanks to the
-XX:PrintFLSStatistics:  
> 
>
========================================================================
========================================================================
======
> {Heap before GC invocations=0 (full 0):
>  par new generation   total 305088K, used 277376K [0x2f800000,
0x43d00000, 0x43d00000)
>   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
>   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
>   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
>  concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000,
0xeb000000, 0xeb000000)
>  concurrent-mark-sweep perm gen total 131072K, used 14913K
[0xeb000000, 0xf3000000, 0xfb000000)
> 8.746: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701235200
> Max   Chunk Size: 701235200
> Number of Blocks: 1
> Av.  Block  Size: 701235200
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 8.746: [ParNew
> Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> - age   1:   15763640 bytes,   15763640 total
> - age   2:         56 bytes,   15763696 total
> : 277376K->15474K(305088K), 0.5657873 secs]
277376K->15474K(3044288K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701218816
> Max   Chunk Size: 701218816
> Number of Blocks: 1
> Av.  Block  Size: 701218816
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
>
========================================================================
========================================================================
======
> 
> Thank you, 
> Dave 
> 
>  David Naas
> Chicago Board Options Exchange 
>  312-786-7222
> 




End of hotspot-gc-dev Digest, Vol 6, Issue 1
********************************************

From neojia at gmail.com  Mon Dec  3 13:12:36 2007
From: neojia at gmail.com (Neo Jia)
Date: Mon, 3 Dec 2007 13:12:36 -0800
Subject: PrintGCStats
In-Reply-To: <8CDFE9CE1B9A344E9AA47C27B7C58DD8168DD7@MSMAIL.cboent.cboe.com>
References: <mailman.1.1196712000.22753.hotspot-gc-dev@openjdk.java.net>
	<8CDFE9CE1B9A344E9AA47C27B7C58DD8168DD7@MSMAIL.cboent.cboe.com>
Message-ID: <5d649bdb0712031312h220dd7bej7fe60c7bf1328d8@mail.gmail.com>

I like the idea about tag/value stuffs.

But I do worry about sending to a separate file unless we put a
timestamp for each log entry, otherwise it would be hard to debug.

Thanks,
Neo

On Dec 3, 2007 12:25 PM, Ciciora, Paul <Ciciora at cboe.com> wrote:
> Since GC formats are constantly changing and different collectors log
> stats differently I'd love to so you adopt a harvestable output format.
> The idea would be to send stats to a separate file and use a name/value
> style format. That way you could send whatever you wanted there and all
> the harvester script or program would have to know is the tag name. The
> name could be terse to save space. You could register the tag names so
> open source changes would conform. If you did that then format changes
> to the human readable output could be separate. You wouldn't have to be
> as sensitive to breaking scripts.
>
>
> Just a thought. I'm probably not the first one to suggest this.
>
> -----Original Message-----
> From: hotspot-gc-dev-bounces at openjdk.java.net
> [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of
> hotspot-gc-dev-request at openjdk.java.net
> Sent: Monday, December 03, 2007 2:00 PM
> To: hotspot-gc-dev at openjdk.java.net
> Subject: hotspot-gc-dev Digest, Vol 6, Issue 1
>
> Send hotspot-gc-dev mailing list submissions to
>         hotspot-gc-dev at openjdk.java.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev
> or, via email, send a message with subject or body 'help' to
>         hotspot-gc-dev-request at openjdk.java.net
>
> You can reach the person managing the list at
>         hotspot-gc-dev-owner at openjdk.java.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of hotspot-gc-dev digest..."
>
>
> Today's Topics:
>
>    1. PrintGCStats (Naas, Dave)
>    2. Re: PrintGCStats (Peter B. Kessler)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 3 Dec 2007 10:57:30 -0600
> From: "Naas, Dave" <naasd at cboe.com>
> Subject: PrintGCStats
> To: <hotspot-gc-dev at openjdk.java.net>
> Message-ID:
>         <4BD174404CF4A34C98322DC926CF862B2289F2 at MSMAIL.cboent.cboe.com>
> Content-Type: text/plain;       charset="iso-8859-1"
>
> Regarding the posting from Oct 26, 2007:
> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
> 4.html
> where Mr. Kessler states:
> quote
>         > > I'll see if we can get an updated version of PrintGCStats
> out to replace the ancient one available at
>         > >
> <http://java.sun.com/developer/technicalArticles/Programming/turbo/>
>         > > That should give you the kinds of statistics you are asking
> for.
>         > >     ... peter
> unquote
>
> I was wondering if a new version of PrintGCStats and PrintGCFixup will
> indeed be made available or if we should proceed with our own updates to
> handle new jdk releases, etc.??
>
> A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics
> flag.  We have found that this setting prevents the current PrintGCStats
> from properly parsing ParNew data and thus doesn't generate the
> Promotion stats (Promo.csv file).   The following snippet shows how the
> ParNew line is now split (shown in bold), thanks to the
> -XX:PrintFLSStatistics:
>
> ========================================================================
> ========================================================================
> ======
> {Heap before GC invocations=0 (full 0):
>  par new generation   total 305088K, used 277376K [0x2f800000,
> 0x43d00000, 0x43d00000)
>   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
>   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
>   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
>  concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000,
> 0xeb000000, 0xeb000000)
>  concurrent-mark-sweep perm gen total 131072K, used 14913K [0xeb000000,
> 0xf3000000, 0xfb000000)
> 8.746: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701235200
> Max   Chunk Size: 701235200
> Number of Blocks: 1
> Av.  Block  Size: 701235200
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 8.746: [ParNew
> Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> - age   1:   15763640 bytes,   15763640 total
> - age   2:         56 bytes,   15763696 total
> : 277376K->15474K(305088K), 0.5657873 secs]
> 277376K->15474K(3044288K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701218816
> Max   Chunk Size: 701218816
> Number of Blocks: 1
> Av.  Block  Size: 701218816
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
> ========================================================================
> ========================================================================
> ======
>
> Thank you,
> Dave
>
>  David Naas
> Chicago Board Options Exchange
>  312-786-7222
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 03 Dec 2007 10:19:41 -0800
> From: "Peter B. Kessler" <Peter.Kessler at Sun.COM>
> Subject: Re: PrintGCStats
> To: "Naas, Dave" <naasd at cboe.com>
> Cc: hotspot-gc-dev at openjdk.java.net
> Message-ID: <475448BD.3030401 at Sun.COM>
> Content-Type: text/plain; format=flowed; charset=ISO-8859-1
>
> It looks like the most recent PrintGCFixup (and PrintGCStats
> and CompareGCStats) is "posted" in
>
>
> http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/51
>
> You have to join all the lines that got wrapped (sigh).
> Sometimes you have to run PrintGCFixup more than once to
> get it to remove things so that it can see other things
> to remove.  Though, in your case, running it twice doesn't
> help remove the output of PrintFLSStatistics.  It does look
> like if you add a cleaner for PrintFLSStatistics, you'll
> have to run PrintGCFixup at least twice, since that output
> gets in the way of the removal of the PrintTenuringDistribution
> flag.
>
> PrintGCFixup and friends look like a prime candidate for putting
> in hotspot/tools/ and letting people make improvements.  If you
> wanted to contribute your addition for removing PrintFLSStatistics,
> I'm sure we could find someone here to sponsor it.
>
>                         ... peter
>
> Naas, Dave wrote:
>
> > Regarding the posting from Oct 26, 2007:
> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
> 4.html
> > where Mr. Kessler states:
> > quote
> >       > > I'll see if we can get an updated version of PrintGCStats
> out to replace the ancient one available at
> >       > >
> <http://java.sun.com/developer/technicalArticles/Programming/turbo/>
> >       > > That should give you the kinds of statistics you are asking
> for.
> >       > >     ... peter
> > unquote
> >
> > I was wondering if a new version of PrintGCStats and PrintGCFixup will
> indeed be made available or if we should proceed with our own updates to
> handle new jdk releases, etc.??
> >
> > A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics
> flag.  We have found that this setting prevents the current PrintGCStats
> from properly parsing ParNew data and thus doesn't generate the
> Promotion stats (Promo.csv file).   The following snippet shows how the
> ParNew line is now split (shown in bold), thanks to the
> -XX:PrintFLSStatistics:
> >
> >
> ========================================================================
> ========================================================================
> ======
> > {Heap before GC invocations=0 (full 0):
> >  par new generation   total 305088K, used 277376K [0x2f800000,
> 0x43d00000, 0x43d00000)
> >   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
> >   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
> >   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
> >  concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000,
> 0xeb000000, 0xeb000000)
> >  concurrent-mark-sweep perm gen total 131072K, used 14913K
> [0xeb000000, 0xf3000000, 0xfb000000)
> > 8.746: [GC Before GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 701235200
> > Max   Chunk Size: 701235200
> > Number of Blocks: 1
> > Av.  Block  Size: 701235200
> > Tree      Height: 1
> > Before GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 0
> > Max   Chunk Size: 0
> > Number of Blocks: 0
> > Tree      Height: 0
> > 8.746: [ParNew
> > Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> > - age   1:   15763640 bytes,   15763640 total
> > - age   2:         56 bytes,   15763696 total
> > : 277376K->15474K(305088K), 0.5657873 secs]
> 277376K->15474K(3044288K)After GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 701218816
> > Max   Chunk Size: 701218816
> > Number of Blocks: 1
> > Av.  Block  Size: 701218816
> > Tree      Height: 1
> > After GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 0
> > Max   Chunk Size: 0
> > Number of Blocks: 0
> > Tree      Height: 0
> > , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
> >
> ========================================================================
> ========================================================================
> ======
> >
> > Thank you,
> > Dave
> >
> >  David Naas
> > Chicago Board Options Exchange
> >  312-786-7222
> >
>
>
>
>
> End of hotspot-gc-dev Digest, Vol 6, Issue 1
> ********************************************
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

From Ciciora at cboe.com  Mon Dec  3 13:16:44 2007
From: Ciciora at cboe.com (Ciciora, Paul)
Date: Mon, 3 Dec 2007 15:16:44 -0600
Subject: PrintGCStats
References: <mailman.1.1196712000.22753.hotspot-gc-dev@openjdk.java.net>
	<8CDFE9CE1B9A344E9AA47C27B7C58DD8168DD7@MSMAIL.cboent.cboe.com>
	<5d649bdb0712031312h220dd7bej7fe60c7bf1328d8@mail.gmail.com>
Message-ID: <8CDFE9CE1B9A344E9AA47C27B7C58DD8168DDF@MSMAIL.cboent.cboe.com>

The idea is you'd include the time stamp as a tagged value. Then you
could have the format you wanted such as seconds since process startup
(my favorite) and each time format would have a separate tag.

-----Original Message-----
From: Neo Jia [mailto:neojia at gmail.com] 
Sent: Monday, December 03, 2007 3:13 PM
To: Ciciora, Paul
Cc: hotspot-gc-dev at openjdk.java.net
Subject: Re: PrintGCStats

I like the idea about tag/value stuffs.

But I do worry about sending to a separate file unless we put a
timestamp for each log entry, otherwise it would be hard to debug.

Thanks,
Neo

On Dec 3, 2007 12:25 PM, Ciciora, Paul <Ciciora at cboe.com> wrote:
> Since GC formats are constantly changing and different collectors log
> stats differently I'd love to so you adopt a harvestable output
format.
> The idea would be to send stats to a separate file and use a
name/value
> style format. That way you could send whatever you wanted there and
all
> the harvester script or program would have to know is the tag name.
The
> name could be terse to save space. You could register the tag names so
> open source changes would conform. If you did that then format changes
> to the human readable output could be separate. You wouldn't have to
be
> as sensitive to breaking scripts.
>
>
> Just a thought. I'm probably not the first one to suggest this.
>
> -----Original Message-----
> From: hotspot-gc-dev-bounces at openjdk.java.net
> [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of
> hotspot-gc-dev-request at openjdk.java.net
> Sent: Monday, December 03, 2007 2:00 PM
> To: hotspot-gc-dev at openjdk.java.net
> Subject: hotspot-gc-dev Digest, Vol 6, Issue 1
>
> Send hotspot-gc-dev mailing list submissions to
>         hotspot-gc-dev at openjdk.java.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev
> or, via email, send a message with subject or body 'help' to
>         hotspot-gc-dev-request at openjdk.java.net
>
> You can reach the person managing the list at
>         hotspot-gc-dev-owner at openjdk.java.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of hotspot-gc-dev digest..."
>
>
> Today's Topics:
>
>    1. PrintGCStats (Naas, Dave)
>    2. Re: PrintGCStats (Peter B. Kessler)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 3 Dec 2007 10:57:30 -0600
> From: "Naas, Dave" <naasd at cboe.com>
> Subject: PrintGCStats
> To: <hotspot-gc-dev at openjdk.java.net>
> Message-ID:
>
<4BD174404CF4A34C98322DC926CF862B2289F2 at MSMAIL.cboent.cboe.com>
> Content-Type: text/plain;       charset="iso-8859-1"
>
> Regarding the posting from Oct 26, 2007:
>
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
> 4.html
> where Mr. Kessler states:
> quote
>         > > I'll see if we can get an updated version of PrintGCStats
> out to replace the ancient one available at
>         > >
> <http://java.sun.com/developer/technicalArticles/Programming/turbo/>
>         > > That should give you the kinds of statistics you are
asking
> for.
>         > >     ... peter
> unquote
>
> I was wondering if a new version of PrintGCStats and PrintGCFixup will
> indeed be made available or if we should proceed with our own updates
to
> handle new jdk releases, etc.??
>
> A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics
> flag.  We have found that this setting prevents the current
PrintGCStats
> from properly parsing ParNew data and thus doesn't generate the
> Promotion stats (Promo.csv file).   The following snippet shows how
the
> ParNew line is now split (shown in bold), thanks to the
> -XX:PrintFLSStatistics:
>
>
========================================================================
>
========================================================================
> ======
> {Heap before GC invocations=0 (full 0):
>  par new generation   total 305088K, used 277376K [0x2f800000,
> 0x43d00000, 0x43d00000)
>   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
>   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
>   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
>  concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000,
> 0xeb000000, 0xeb000000)
>  concurrent-mark-sweep perm gen total 131072K, used 14913K
[0xeb000000,
> 0xf3000000, 0xfb000000)
> 8.746: [GC Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701235200
> Max   Chunk Size: 701235200
> Number of Blocks: 1
> Av.  Block  Size: 701235200
> Tree      Height: 1
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> 8.746: [ParNew
> Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> - age   1:   15763640 bytes,   15763640 total
> - age   2:         56 bytes,   15763696 total
> : 277376K->15474K(305088K), 0.5657873 secs]
> 277376K->15474K(3044288K)After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 701218816
> Max   Chunk Size: 701218816
> Number of Blocks: 1
> Av.  Block  Size: 701218816
> Tree      Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max   Chunk Size: 0
> Number of Blocks: 0
> Tree      Height: 0
> , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
>
========================================================================
>
========================================================================
> ======
>
> Thank you,
> Dave
>
>  David Naas
> Chicago Board Options Exchange
>  312-786-7222
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 03 Dec 2007 10:19:41 -0800
> From: "Peter B. Kessler" <Peter.Kessler at Sun.COM>
> Subject: Re: PrintGCStats
> To: "Naas, Dave" <naasd at cboe.com>
> Cc: hotspot-gc-dev at openjdk.java.net
> Message-ID: <475448BD.3030401 at Sun.COM>
> Content-Type: text/plain; format=flowed; charset=ISO-8859-1
>
> It looks like the most recent PrintGCFixup (and PrintGCStats
> and CompareGCStats) is "posted" in
>
>
> http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/51
>
> You have to join all the lines that got wrapped (sigh).
> Sometimes you have to run PrintGCFixup more than once to
> get it to remove things so that it can see other things
> to remove.  Though, in your case, running it twice doesn't
> help remove the output of PrintFLSStatistics.  It does look
> like if you add a cleaner for PrintFLSStatistics, you'll
> have to run PrintGCFixup at least twice, since that output
> gets in the way of the removal of the PrintTenuringDistribution
> flag.
>
> PrintGCFixup and friends look like a prime candidate for putting
> in hotspot/tools/ and letting people make improvements.  If you
> wanted to contribute your addition for removing PrintFLSStatistics,
> I'm sure we could find someone here to sponsor it.
>
>                         ... peter
>
> Naas, Dave wrote:
>
> > Regarding the posting from Oct 26, 2007:
>
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
> 4.html
> > where Mr. Kessler states:
> > quote
> >       > > I'll see if we can get an updated version of PrintGCStats
> out to replace the ancient one available at
> >       > >
> <http://java.sun.com/developer/technicalArticles/Programming/turbo/>
> >       > > That should give you the kinds of statistics you are
asking
> for.
> >       > >     ... peter
> > unquote
> >
> > I was wondering if a new version of PrintGCStats and PrintGCFixup
will
> indeed be made available or if we should proceed with our own updates
to
> handle new jdk releases, etc.??
> >
> > A specific, recent example is jdk 1.6 with the
-XX:PrintFLSStatistics
> flag.  We have found that this setting prevents the current
PrintGCStats
> from properly parsing ParNew data and thus doesn't generate the
> Promotion stats (Promo.csv file).   The following snippet shows how
the
> ParNew line is now split (shown in bold), thanks to the
> -XX:PrintFLSStatistics:
> >
> >
>
========================================================================
>
========================================================================
> ======
> > {Heap before GC invocations=0 (full 0):
> >  par new generation   total 305088K, used 277376K [0x2f800000,
> 0x43d00000, 0x43d00000)
> >   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
> >   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
> >   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
> >  concurrent mark-sweep generation total 2739200K, used 0K
[0x43d00000,
> 0xeb000000, 0xeb000000)
> >  concurrent-mark-sweep perm gen total 131072K, used 14913K
> [0xeb000000, 0xf3000000, 0xfb000000)
> > 8.746: [GC Before GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 701235200
> > Max   Chunk Size: 701235200
> > Number of Blocks: 1
> > Av.  Block  Size: 701235200
> > Tree      Height: 1
> > Before GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 0
> > Max   Chunk Size: 0
> > Number of Blocks: 0
> > Tree      Height: 0
> > 8.746: [ParNew
> > Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> > - age   1:   15763640 bytes,   15763640 total
> > - age   2:         56 bytes,   15763696 total
> > : 277376K->15474K(305088K), 0.5657873 secs]
> 277376K->15474K(3044288K)After GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 701218816
> > Max   Chunk Size: 701218816
> > Number of Blocks: 1
> > Av.  Block  Size: 701218816
> > Tree      Height: 1
> > After GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 0
> > Max   Chunk Size: 0
> > Number of Blocks: 0
> > Tree      Height: 0
> > , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
> >
>
========================================================================
>
========================================================================
> ======
> >
> > Thank you,
> > Dave
> >
> >  David Naas
> > Chicago Board Options Exchange
> >  312-786-7222
> >
>
>
>
>
> End of hotspot-gc-dev Digest, Vol 6, Issue 1
> ********************************************
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

From neojia at gmail.com  Mon Dec  3 13:21:04 2007
From: neojia at gmail.com (Neo Jia)
Date: Mon, 3 Dec 2007 13:21:04 -0800
Subject: PrintGCStats
In-Reply-To: <8CDFE9CE1B9A344E9AA47C27B7C58DD8168DDF@MSMAIL.cboent.cboe.com>
References: <mailman.1.1196712000.22753.hotspot-gc-dev@openjdk.java.net>
	<8CDFE9CE1B9A344E9AA47C27B7C58DD8168DD7@MSMAIL.cboent.cboe.com>
	<5d649bdb0712031312h220dd7bej7fe60c7bf1328d8@mail.gmail.com>
	<8CDFE9CE1B9A344E9AA47C27B7C58DD8168DDF@MSMAIL.cboent.cboe.com>
Message-ID: <5d649bdb0712031321o1d37c6f7w4921063560b5e8f9@mail.gmail.com>

On Dec 3, 2007 1:16 PM, Ciciora, Paul <Ciciora at cboe.com> wrote:
> The idea is you'd include the time stamp as a tagged value. Then you
> could have the format you wanted such as seconds since process startup
> (my favorite) and each time format would have a separate tag.

But those stamps are only included in GC logs, right?

Thanks,
Neo

>
> -----Original Message-----
> From: Neo Jia [mailto:neojia at gmail.com]
> Sent: Monday, December 03, 2007 3:13 PM
> To: Ciciora, Paul
> Cc: hotspot-gc-dev at openjdk.java.net
>
> Subject: Re: PrintGCStats
>
> I like the idea about tag/value stuffs.
>
> But I do worry about sending to a separate file unless we put a
> timestamp for each log entry, otherwise it would be hard to debug.
>
> Thanks,
> Neo
>
> On Dec 3, 2007 12:25 PM, Ciciora, Paul <Ciciora at cboe.com> wrote:
> > Since GC formats are constantly changing and different collectors log
> > stats differently I'd love to so you adopt a harvestable output
> format.
> > The idea would be to send stats to a separate file and use a
> name/value
> > style format. That way you could send whatever you wanted there and
> all
> > the harvester script or program would have to know is the tag name.
> The
> > name could be terse to save space. You could register the tag names so
> > open source changes would conform. If you did that then format changes
> > to the human readable output could be separate. You wouldn't have to
> be
> > as sensitive to breaking scripts.
> >
> >
> > Just a thought. I'm probably not the first one to suggest this.
> >
> > -----Original Message-----
> > From: hotspot-gc-dev-bounces at openjdk.java.net
> > [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of
> > hotspot-gc-dev-request at openjdk.java.net
> > Sent: Monday, December 03, 2007 2:00 PM
> > To: hotspot-gc-dev at openjdk.java.net
> > Subject: hotspot-gc-dev Digest, Vol 6, Issue 1
> >
> > Send hotspot-gc-dev mailing list submissions to
> >         hotspot-gc-dev at openjdk.java.net
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev
> > or, via email, send a message with subject or body 'help' to
> >         hotspot-gc-dev-request at openjdk.java.net
> >
> > You can reach the person managing the list at
> >         hotspot-gc-dev-owner at openjdk.java.net
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of hotspot-gc-dev digest..."
> >
> >
> > Today's Topics:
> >
> >    1. PrintGCStats (Naas, Dave)
> >    2. Re: PrintGCStats (Peter B. Kessler)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Mon, 3 Dec 2007 10:57:30 -0600
> > From: "Naas, Dave" <naasd at cboe.com>
> > Subject: PrintGCStats
> > To: <hotspot-gc-dev at openjdk.java.net>
> > Message-ID:
> >
> <4BD174404CF4A34C98322DC926CF862B2289F2 at MSMAIL.cboent.cboe.com>
> > Content-Type: text/plain;       charset="iso-8859-1"
> >
> > Regarding the posting from Oct 26, 2007:
> >
> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
> > 4.html
> > where Mr. Kessler states:
> > quote
> >         > > I'll see if we can get an updated version of PrintGCStats
> > out to replace the ancient one available at
> >         > >
> > <http://java.sun.com/developer/technicalArticles/Programming/turbo/>
> >         > > That should give you the kinds of statistics you are
> asking
> > for.
> >         > >     ... peter
> > unquote
> >
> > I was wondering if a new version of PrintGCStats and PrintGCFixup will
> > indeed be made available or if we should proceed with our own updates
> to
> > handle new jdk releases, etc.??
> >
> > A specific, recent example is jdk 1.6 with the -XX:PrintFLSStatistics
> > flag.  We have found that this setting prevents the current
> PrintGCStats
> > from properly parsing ParNew data and thus doesn't generate the
> > Promotion stats (Promo.csv file).   The following snippet shows how
> the
> > ParNew line is now split (shown in bold), thanks to the
> > -XX:PrintFLSStatistics:
> >
> >
> ========================================================================
> >
> ========================================================================
> > ======
> > {Heap before GC invocations=0 (full 0):
> >  par new generation   total 305088K, used 277376K [0x2f800000,
> > 0x43d00000, 0x43d00000)
> >   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
> >   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
> >   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
> >  concurrent mark-sweep generation total 2739200K, used 0K [0x43d00000,
> > 0xeb000000, 0xeb000000)
> >  concurrent-mark-sweep perm gen total 131072K, used 14913K
> [0xeb000000,
> > 0xf3000000, 0xfb000000)
> > 8.746: [GC Before GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 701235200
> > Max   Chunk Size: 701235200
> > Number of Blocks: 1
> > Av.  Block  Size: 701235200
> > Tree      Height: 1
> > Before GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 0
> > Max   Chunk Size: 0
> > Number of Blocks: 0
> > Tree      Height: 0
> > 8.746: [ParNew
> > Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> > - age   1:   15763640 bytes,   15763640 total
> > - age   2:         56 bytes,   15763696 total
> > : 277376K->15474K(305088K), 0.5657873 secs]
> > 277376K->15474K(3044288K)After GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 701218816
> > Max   Chunk Size: 701218816
> > Number of Blocks: 1
> > Av.  Block  Size: 701218816
> > Tree      Height: 1
> > After GC:
> > Statistics for BinaryTreeDictionary:
> > ------------------------------------
> > Total Free Space: 0
> > Max   Chunk Size: 0
> > Number of Blocks: 0
> > Tree      Height: 0
> > , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
> >
> ========================================================================
> >
> ========================================================================
> > ======
> >
> > Thank you,
> > Dave
> >
> >  David Naas
> > Chicago Board Options Exchange
> >  312-786-7222
> >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Mon, 03 Dec 2007 10:19:41 -0800
> > From: "Peter B. Kessler" <Peter.Kessler at Sun.COM>
> > Subject: Re: PrintGCStats
> > To: "Naas, Dave" <naasd at cboe.com>
> > Cc: hotspot-gc-dev at openjdk.java.net
> > Message-ID: <475448BD.3030401 at Sun.COM>
> > Content-Type: text/plain; format=flowed; charset=ISO-8859-1
> >
> > It looks like the most recent PrintGCFixup (and PrintGCStats
> > and CompareGCStats) is "posted" in
> >
> >
> > http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/51
> >
> > You have to join all the lines that got wrapped (sigh).
> > Sometimes you have to run PrintGCFixup more than once to
> > get it to remove things so that it can see other things
> > to remove.  Though, in your case, running it twice doesn't
> > help remove the output of PrintFLSStatistics.  It does look
> > like if you add a cleaner for PrintFLSStatistics, you'll
> > have to run PrintGCFixup at least twice, since that output
> > gets in the way of the removal of the PrintTenuringDistribution
> > flag.
> >
> > PrintGCFixup and friends look like a prime candidate for putting
> > in hotspot/tools/ and letting people make improvements.  If you
> > wanted to contribute your addition for removing PrintFLSStatistics,
> > I'm sure we could find someone here to sponsor it.
> >
> >                         ... peter
> >
> > Naas, Dave wrote:
> >
> > > Regarding the posting from Oct 26, 2007:
> >
> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-October/00006
> > 4.html
> > > where Mr. Kessler states:
> > > quote
> > >       > > I'll see if we can get an updated version of PrintGCStats
> > out to replace the ancient one available at
> > >       > >
> > <http://java.sun.com/developer/technicalArticles/Programming/turbo/>
> > >       > > That should give you the kinds of statistics you are
> asking
> > for.
> > >       > >     ... peter
> > > unquote
> > >
> > > I was wondering if a new version of PrintGCStats and PrintGCFixup
> will
> > indeed be made available or if we should proceed with our own updates
> to
> > handle new jdk releases, etc.??
> > >
> > > A specific, recent example is jdk 1.6 with the
> -XX:PrintFLSStatistics
> > flag.  We have found that this setting prevents the current
> PrintGCStats
> > from properly parsing ParNew data and thus doesn't generate the
> > Promotion stats (Promo.csv file).   The following snippet shows how
> the
> > ParNew line is now split (shown in bold), thanks to the
> > -XX:PrintFLSStatistics:
> > >
> > >
> >
> ========================================================================
> >
> ========================================================================
> > ======
> > > {Heap before GC invocations=0 (full 0):
> > >  par new generation   total 305088K, used 277376K [0x2f800000,
> > 0x43d00000, 0x43d00000)
> > >   eden space 277376K, 100% used [0x2f800000, 0x406e0000, 0x406e0000)
> > >   from space 27712K,   0% used [0x406e0000, 0x406e0000, 0x421f0000)
> > >   to   space 27712K,   0% used [0x421f0000, 0x421f0000, 0x43d00000)
> > >  concurrent mark-sweep generation total 2739200K, used 0K
> [0x43d00000,
> > 0xeb000000, 0xeb000000)
> > >  concurrent-mark-sweep perm gen total 131072K, used 14913K
> > [0xeb000000, 0xf3000000, 0xfb000000)
> > > 8.746: [GC Before GC:
> > > Statistics for BinaryTreeDictionary:
> > > ------------------------------------
> > > Total Free Space: 701235200
> > > Max   Chunk Size: 701235200
> > > Number of Blocks: 1
> > > Av.  Block  Size: 701235200
> > > Tree      Height: 1
> > > Before GC:
> > > Statistics for BinaryTreeDictionary:
> > > ------------------------------------
> > > Total Free Space: 0
> > > Max   Chunk Size: 0
> > > Number of Blocks: 0
> > > Tree      Height: 0
> > > 8.746: [ParNew
> > > Desired survivor size 14188544 bytes, new threshold 1 (max 4)
> > > - age   1:   15763640 bytes,   15763640 total
> > > - age   2:         56 bytes,   15763696 total
> > > : 277376K->15474K(305088K), 0.5657873 secs]
> > 277376K->15474K(3044288K)After GC:
> > > Statistics for BinaryTreeDictionary:
> > > ------------------------------------
> > > Total Free Space: 701218816
> > > Max   Chunk Size: 701218816
> > > Number of Blocks: 1
> > > Av.  Block  Size: 701218816
> > > Tree      Height: 1
> > > After GC:
> > > Statistics for BinaryTreeDictionary:
> > > ------------------------------------
> > > Total Free Space: 0
> > > Max   Chunk Size: 0
> > > Number of Blocks: 0
> > > Tree      Height: 0
> > > , 0.5668866 secs] [Times: user=0.34 sys=1.07, real=0.57 secs]
> > >
> >
> ========================================================================
> >
> ========================================================================
> > ======
> > >
> > > Thank you,
> > > Dave
> > >
> > >  David Naas
> > > Chicago Board Options Exchange
> > >  312-786-7222
> > >
> >
> >
> >
> >
> > End of hotspot-gc-dev Digest, Vol 6, Issue 1
> > ********************************************
> >
>
>
>
> --
> I would remember that if researchers were not ambitious
> probably today we haven't the technology we are using!
>



-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

From Jon.Masamitsu at Sun.COM  Thu Dec  6 07:24:38 2007
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 06 Dec 2007 07:24:38 -0800
Subject: Please ignore
Message-ID: <47581436.9010108@Sun.COM>



hotspot-gc-use at openjdk.java.net is forwarded to
hotspot-gc-dev at openjdk.java.net without moderation?
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From Jon.Masamitsu at Sun.COM  Thu Dec  6 10:47:48 2007
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 06 Dec 2007 10:47:48 -0800
Subject: Please ignore
Message-ID: <475843D4.7060101@Sun.COM>


hotspot-gc-use at openjdk.java.net is forwarded to
hotspot-gc-dev at openjdk.java.net without moderation?


On hotspot-gc-dev who is the sender?
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From Jon.Masamitsu at Sun.COM  Thu Dec  6 11:51:52 2007
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 06 Dec 2007 11:51:52 -0800
Subject: Please ignore in hotspot-gc-dev too
Message-ID: <475852D8.8000500@Sun.COM>


hotspot-gc-use at openjdk.java.net is forwarded to
hotspot-gc-dev at openjdk.java.net without moderation?


On hotspot-gc-dev who is the sender?

hotspot-gc-use at openjdk.java.net was added to the list
of acceptable aliases.

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From Y.S.Ramakrishna at Sun.COM  Fri Dec  7 16:32:24 2007
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Fri, 07 Dec 2007 16:32:24 -0800
Subject: request for review: 6634032 (medium-to-small): CMS: Need
	CMSInitiatingPermOccupancyFraction for perm,
	divorcing from CMSInitiatingOccupancyFraction
Message-ID: <fc8ea97d6b9c.47597598@sun.com>


[Attached is a patch of diffs as well as a webrev tar-ball, whichever format you
prefer. The latter may be easier for the more extensive changes in the case
of one of the files at least, where the diffs may get a bit hairy to read.
Sun-internal reviewers can access the webrev via a pointer in the suggested
fix section of the bug report.]

6634032 CMS: Need CMSInitiatingPermOccupancyFraction for perm, divorcing from CMSInitiatingOccupancyFraction

In addition, we can now dynamically decide at the start of a CMS collection
cycle whether we will be collecting the perm gen in that cycle. I introduced
a couple of experimental knobs to tweak that decision policy, but left the
default current policy intact (modulo the divorcing of the two collection thresholds).
See the block comment preceding update_should_unload_classes().
The settings of the knobs will likely change based on further experiments
and experience, but the current settings are so as to rock the boat as
little as possible. This should fix the problems reported by Anuj Lal
and Thomas Viessmann on this mailing list. (vide the following
thread: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-November/000081.html )

Further improvements are planned in CR 6543076 "Adaptive collection of perm gen",
leveraging the infrastructure here to allow dynamic per-cycle decisions based on
perm gen allocation rate estimates etc.

Fix Verified: yes

Verification Testing: +PrintGCDetails was used along with default
   and various non-default settings to verify that perm gen collections
   were occurring as designed.

Other Testing: on-going internal testing (rwl and jprt); performance data will
   be collected and shared prior to any push.

PS: mixed in with the diffs is an old diff for:-
6621144 CMS: assertion failure "is_cms_thread == Thread::current()->is_ConcurrentGC_thread()"
which you can also review (or feel free to ignore for the purposes of this
code review).

Many thanks for your reviews, feedback, comments and suggestions.
-- ramki
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cms_bugs.patch
Type: text/x-patch
Size: 29932 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20071207/a075b50a/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: webrev.tar.gz
Type: application/x-gzip
Size: 1692996 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20071207/a075b50a/attachment-0003.bin 

From Keith.Holdaway at sas.com  Mon Dec 10 10:16:05 2007
From: Keith.Holdaway at sas.com (Keith Holdaway)
Date: Mon, 10 Dec 2007 13:16:05 -0500
Subject: 64 bit CMS JDK 5.0 u14
In-Reply-To: <fc8ea97d6b9c.47597598@sun.com>
References: <fc8ea97d6b9c.47597598@sun.com>
Message-ID: <304E9E55F6A4BE4B910C2437D4D1B49608FB48C825@MERCMBX14.na.sas.com>

Gentlemen,

I am running some midle-tier Portal load tests in WLS 9.2 MP2 with Sun JDK 5.0 u14. I am running 100 concurrent users that logon, navigate, open portlets, and eventually logoff; only to logon again and repeat the cycle. My testing people have establish Load Runner scripts to put the Portal software through an endurance test over 5 days with the 100 users.

Normally on 32-bit WLS 8.1 SP6 with JDK 1.4.2_13 we run with the following VM args; we attain some 3.4 million passed transactions with zero failed transactions:

-server -Xms1400m -Xmx1400m -XX:NewSize=64m -XX:MaxNewSize=64m -XX:PermSize=128m -XX:MaxPermSize=128m -Xss128k -XX:-UseTLAB -XX:+DisableExplicitGC
-Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true

CMS never works very well in the 32-bit environment; failing miserably above; although at JDK 1.4.2_15, we see some 2 million passed transactions with 1130+ failed transactions owing to 120 seconds timeouts in concurrent mode failures.

Now, in the 64 bit environment running on AMD Windows Server 2003, I can run pretty successfully with CMS:

-Xms1500m -Xmx3500m -XX:NewSize=320m -XX:MaxNewSize=320m -Xss256k -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=40 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -verbosegc -Xloggc:C:\keith\GCLogs\gc11.txt

I can achieve some 3.4 million as for the throughput collector in 32 bit env.

But, I do note several thousand failed transactions that correlate with concurrent mode failures after some 24 hrs; pauses in the 400-600 seconds range when Full GC takes over.

I have tried varying the CMSInitiatingOccupancyFraction to 20%, but the CMS mode failures still occur.

I am now running with the incremental mode CMS; but anticipate further very long pauses.

The VM always recovers very well after these sporadic Full GCs, but to eradicate them, should I run with an 8 GB heap or something along those lines.? I also read something about killing the swap file?

My AMD 64 bit bx, unfortunately for now is restricted to 4 GB RAM; but I am adding a further 4 GB soon. I am about to go to the Solaris SPARC 64 bit and run the exact same scenario with a 7-8 GB heap.

I read about the occupancy fration for OG and Perm Gen; do I need to apply this patch. Our Perm Gen is always set to 128 MB and only ever attains 108 MB.

Any feedback would help us in our endeavours to support our EBI apps in a 64 bit env.

keith


Keith R Holdaway
Java Development Technologies

SAS...  The Power to Know

Carpe Diem ...


From Y.S.Ramakrishna at Sun.COM  Mon Dec 10 10:44:48 2007
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Mon, 10 Dec 2007 10:44:48 -0800
Subject: 64 bit CMS JDK 5.0 u14
In-Reply-To: <304E9E55F6A4BE4B910C2437D4D1B49608FB48C825@MERCMBX14.na.sas.com>
References: <fc8ea97d6b9c.47597598@sun.com>
	<304E9E55F6A4BE4B910C2437D4D1B49608FB48C825@MERCMBX14.na.sas.com>
Message-ID: <475D8920.8000501@Sun.COM>

Hi Keith --

> I am running some midle-tier Portal load tests in WLS 9.2 MP2 with Sun JDK 5.0 u14. I am running 100 concurrent users that logon, navigate, open portlets, and eventually logoff; only to logon again and repeat the cycle. My testing people have establish Load Runner scripts to put the Portal software through an endurance test over 5 days with the 100 users.
> 
> Normally on 32-bit WLS 8.1 SP6 with JDK 1.4.2_13 we run with the following VM args; we attain some 3.4 million passed transactions with zero failed transactions:
> 
> -server -Xms1400m -Xmx1400m -XX:NewSize=64m -XX:MaxNewSize=64m -XX:PermSize=128m -XX:MaxPermSize=128m -Xss128k -XX:-UseTLAB -XX:+DisableExplicitGC
> -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true
> 
> CMS never works very well in the 32-bit environment; failing miserably above; although at JDK 1.4.2_15, we see some 2 million passed transactions with 1130+ failed transactions owing to 120 seconds timeouts in concurrent mode failures.
> 
> Now, in the 64 bit environment running on AMD Windows Server 2003, I can run pretty successfully with CMS:
> 
> -Xms1500m -Xmx3500m -XX:NewSize=320m -XX:MaxNewSize=320m -Xss256k -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=40 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -verbosegc -Xloggc:C:\keith\GCLogs\gc11.txt
> 
> I can achieve some 3.4 million as for the throughput collector in 32 bit env.
> 
> But, I do note several thousand failed transactions that correlate with concurrent mode failures after some 24 hrs; pauses in the 400-600 seconds range when Full GC takes over.
> 

The fact that you start seeing the concurrent mode failures after
24 hours indicates to me strongly that the old generation gets slowly
fragmented over a period of time. (Recall that the CMS collector is
non-moving.)

Can you confirm that the heap occupancy itself is constant (or nearly
so) following CMS collection cycles, and that the full gc that follows
a concurrent mode failure does not unload classes? Recall that CMS will
not, by default, unload classes during concurrent cycles unless
explicitly instructed to do so via:

      -XX:+CMSClassUnloadingEnabled -XX:+PermGenSweepingEnabled

(the second option is needed in pre-6.0 JVM's, but not in more
recent JVM's).

> I have tried varying the CMSInitiatingOccupancyFraction to 20%, but the CMS mode failures still occur.

It is usually a good idea to use survivor spaces to both reduce the
pressure on the concurrent collector (by promoting less to the old
gen), but also to reduce the spread in object sizes and lifetimes
of the objects that do get promoted. I'd suggest using survivor
spaces to make sure that survivors stay in the young gen for at least
one scavenge (MaxTenuringThreshold = 1, possibly more, as experiments
dictate), possibly more. A downside is possibly longer scavenges,
but consider that the price for (possibly) avoiding concurrent
mode failure.

Prematurely promoting objects (besides the two points made above),
can also reduce floating garbage and reduce CMS remark pause
times (by reducing mutation rates in the old generation).

> 
> I am now running with the incremental mode CMS; but anticipate further very long pauses.

 From what you described above (running CMS all the time by setting the
initiation threshold very low), it does not look as though iCMS will
buy you anything.

> 
> The VM always recovers very well after these sporadic Full GCs, but to eradicate them, should I run with an 8 GB heap or something along those lines.? I also read something about killing the swap file?
> 
> My AMD 64 bit bx, unfortunately for now is restricted to 4 GB RAM; but I am adding a further 4 GB soon. I am about to go to the Solaris SPARC 64 bit and run the exact same scenario with a 7-8 GB heap.

Increasing the heap size can indeed sometimes help you avoid
concurrent mode failure from fragmentation. (But first make
sure to enable survivor spaces and, if applicable, perm gen
collection.)

> 
> I read about the occupancy fration for OG and Perm Gen; do I need to apply this patch. Our Perm Gen is always set to 128 MB and only ever attains 108 MB.

The webrev i posted late last week should not really apply directly to
your case (except inasmuch as, in the event that you enable perm gen
collection, it might allow you to get away with not collecting the
perm gen per each cycle, and thus help keep cms remark pauses possibly
shorter). I would not worry about this patch at the level at which you
are tuning currently (which is mainly looking to avoid the concurrent
mode failures).

-- ramki

> 
> Any feedback would help us in our endeavours to support our EBI apps in a 64 bit env.
> 
> keith
> 
> 
> Keith R Holdaway
> Java Development Technologies
> 
> SAS...  The Power to Know
> 
> Carpe Diem ...
> 


From Y.S.Ramakrishna at Sun.COM  Mon Dec 10 10:57:33 2007
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Mon, 10 Dec 2007 10:57:33 -0800
Subject: 64 bit CMS JDK 5.0 u14
In-Reply-To: <475D8920.8000501@Sun.COM>
References: <fc8ea97d6b9c.47597598@sun.com>
	<304E9E55F6A4BE4B910C2437D4D1B49608FB48C825@MERCMBX14.na.sas.com>
	<475D8920.8000501@Sun.COM>
Message-ID: <475D8C1D.5040406@Sun.COM>

Hi Keith --

Jon also has talks about this and related
issues in a few of his blogs; see:-

http://blogs.sun.com/jonthecollector/date/20060413
http://blogs.sun.com/jonthecollector/date/20060306
http://blogs.sun.com/jonthecollector/date/20060404
http://blogs.sun.com/jonthecollector/date/20070622

-- ramki

Y.S.Ramakrishna at Sun.COM wrote:
> Hi Keith --
> 
>> I am running some midle-tier Portal load tests in WLS 9.2 MP2 with Sun 
>> JDK 5.0 u14. I am running 100 concurrent users that logon, navigate, 
>> open portlets, and eventually logoff; only to logon again and repeat 
>> the cycle. My testing people have establish Load Runner scripts to put 
>> the Portal software through an endurance test over 5 days with the 100 
>> users.
>>
>> Normally on 32-bit WLS 8.1 SP6 with JDK 1.4.2_13 we run with the 
>> following VM args; we attain some 3.4 million passed transactions with 
>> zero failed transactions:
>>
>> -server -Xms1400m -Xmx1400m -XX:NewSize=64m -XX:MaxNewSize=64m 
>> -XX:PermSize=128m -XX:MaxPermSize=128m -Xss128k -XX:-UseTLAB 
>> -XX:+DisableExplicitGC
>> -Dsun.rmi.dgc.client.gcInterval=3600000 
>> -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true
>>
>> CMS never works very well in the 32-bit environment; failing miserably 
>> above; although at JDK 1.4.2_15, we see some 2 million passed 
>> transactions with 1130+ failed transactions owing to 120 seconds 
>> timeouts in concurrent mode failures.
>>
>> Now, in the 64 bit environment running on AMD Windows Server 2003, I 
>> can run pretty successfully with CMS:
>>
>> -Xms1500m -Xmx3500m -XX:NewSize=320m -XX:MaxNewSize=320m -Xss256k 
>> -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC 
>> -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly 
>> -XX:CMSInitiatingOccupancyFraction=40 
>> -Dsun.rmi.dgc.client.gcInterval=3600000 
>> -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true 
>> -Dcom.sun.management.jmxremote -verbosegc 
>> -Xloggc:C:\keith\GCLogs\gc11.txt
>>
>> I can achieve some 3.4 million as for the throughput collector in 32 
>> bit env.
>>
>> But, I do note several thousand failed transactions that correlate 
>> with concurrent mode failures after some 24 hrs; pauses in the 400-600 
>> seconds range when Full GC takes over.
>>
> 
> The fact that you start seeing the concurrent mode failures after
> 24 hours indicates to me strongly that the old generation gets slowly
> fragmented over a period of time. (Recall that the CMS collector is
> non-moving.)
> 
> Can you confirm that the heap occupancy itself is constant (or nearly
> so) following CMS collection cycles, and that the full gc that follows
> a concurrent mode failure does not unload classes? Recall that CMS will
> not, by default, unload classes during concurrent cycles unless
> explicitly instructed to do so via:
> 
>      -XX:+CMSClassUnloadingEnabled -XX:+PermGenSweepingEnabled
> 
> (the second option is needed in pre-6.0 JVM's, but not in more
> recent JVM's).
> 
>> I have tried varying the CMSInitiatingOccupancyFraction to 20%, but 
>> the CMS mode failures still occur.
> 
> It is usually a good idea to use survivor spaces to both reduce the
> pressure on the concurrent collector (by promoting less to the old
> gen), but also to reduce the spread in object sizes and lifetimes
> of the objects that do get promoted. I'd suggest using survivor
> spaces to make sure that survivors stay in the young gen for at least
> one scavenge (MaxTenuringThreshold = 1, possibly more, as experiments
> dictate), possibly more. A downside is possibly longer scavenges,
> but consider that the price for (possibly) avoiding concurrent
> mode failure.
> 
> Prematurely promoting objects (besides the two points made above),
> can also reduce floating garbage and reduce CMS remark pause
> times (by reducing mutation rates in the old generation).
> 
>>
>> I am now running with the incremental mode CMS; but anticipate further 
>> very long pauses.
> 
>  From what you described above (running CMS all the time by setting the
> initiation threshold very low), it does not look as though iCMS will
> buy you anything.
> 
>>
>> The VM always recovers very well after these sporadic Full GCs, but to 
>> eradicate them, should I run with an 8 GB heap or something along 
>> those lines.? I also read something about killing the swap file?
>>
>> My AMD 64 bit bx, unfortunately for now is restricted to 4 GB RAM; but 
>> I am adding a further 4 GB soon. I am about to go to the Solaris SPARC 
>> 64 bit and run the exact same scenario with a 7-8 GB heap.
> 
> Increasing the heap size can indeed sometimes help you avoid
> concurrent mode failure from fragmentation. (But first make
> sure to enable survivor spaces and, if applicable, perm gen
> collection.)
> 
>>
>> I read about the occupancy fration for OG and Perm Gen; do I need to 
>> apply this patch. Our Perm Gen is always set to 128 MB and only ever 
>> attains 108 MB.
> 
> The webrev i posted late last week should not really apply directly to
> your case (except inasmuch as, in the event that you enable perm gen
> collection, it might allow you to get away with not collecting the
> perm gen per each cycle, and thus help keep cms remark pauses possibly
> shorter). I would not worry about this patch at the level at which you
> are tuning currently (which is mainly looking to avoid the concurrent
> mode failures).
> 
> -- ramki
> 
>>
>> Any feedback would help us in our endeavours to support our EBI apps 
>> in a 64 bit env.
>>
>> keith
>>
>>
>> Keith R Holdaway
>> Java Development Technologies
>>
>> SAS...  The Power to Know
>>
>> Carpe Diem ...
>>
> 
> 


From tony.printezis at sun.com  Mon Dec 10 10:59:01 2007
From: tony.printezis at sun.com (Tony Printezis)
Date: Mon, 10 Dec 2007 13:59:01 -0500
Subject: 64 bit CMS JDK 5.0 u14
In-Reply-To: <475D8920.8000501@Sun.COM>
References: <fc8ea97d6b9c.47597598@sun.com>
	<304E9E55F6A4BE4B910C2437D4D1B49608FB48C825@MERCMBX14.na.sas.com>
	<475D8920.8000501@Sun.COM>
Message-ID: <475D8C75.5010509@sun.com>

Keith and Ramki,

I'd like to add a couple of things to Ramki's informative reply.

First, I wonder why the Portal ran fine with the default GC in 1.4.2 
with no failed (I assumed timed out, right?) transactions. The default 
(aka serial) GC is not a low-pause GC, so maybe the pause times it gave 
you were good enough. BTW, what are your latency targets? hundreds of 
ms? seconds?

Second, we'd be able to help you a bit more constructively if you gave 
us some GC logs to look at (and if you have both WLS 8.1 and 9.2 logs, 
it'd be nice to get them so that we can compare them).

 > CMS never works very well in the 32-bit environment

FWIW, a lot of our customers are happy with CMS in a 32-bit environment! :-)

Tony

Y.S.Ramakrishna at sun.com wrote:
> Hi Keith --
>
>> I am running some midle-tier Portal load tests in WLS 9.2 MP2 with 
>> Sun JDK 5.0 u14. I am running 100 concurrent users that logon, 
>> navigate, open portlets, and eventually logoff; only to logon again 
>> and repeat the cycle. My testing people have establish Load Runner 
>> scripts to put the Portal software through an endurance test over 5 
>> days with the 100 users.
>>
>> Normally on 32-bit WLS 8.1 SP6 with JDK 1.4.2_13 we run with the 
>> following VM args; we attain some 3.4 million passed transactions 
>> with zero failed transactions:
>>
>> -server -Xms1400m -Xmx1400m -XX:NewSize=64m -XX:MaxNewSize=64m 
>> -XX:PermSize=128m -XX:MaxPermSize=128m -Xss128k -XX:-UseTLAB 
>> -XX:+DisableExplicitGC
>> -Dsun.rmi.dgc.client.gcInterval=3600000 
>> -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true
>>
>> CMS never works very well in the 32-bit environment; failing 
>> miserably above; although at JDK 1.4.2_15, we see some 2 million 
>> passed transactions with 1130+ failed transactions owing to 120 
>> seconds timeouts in concurrent mode failures.
>>
>> Now, in the 64 bit environment running on AMD Windows Server 2003, I 
>> can run pretty successfully with CMS:
>>
>> -Xms1500m -Xmx3500m -XX:NewSize=320m -XX:MaxNewSize=320m -Xss256k 
>> -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC 
>> -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly 
>> -XX:CMSInitiatingOccupancyFraction=40 
>> -Dsun.rmi.dgc.client.gcInterval=3600000 
>> -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true 
>> -Dcom.sun.management.jmxremote -verbosegc 
>> -Xloggc:C:\keith\GCLogs\gc11.txt
>>
>> I can achieve some 3.4 million as for the throughput collector in 32 
>> bit env.
>>
>> But, I do note several thousand failed transactions that correlate 
>> with concurrent mode failures after some 24 hrs; pauses in the 
>> 400-600 seconds range when Full GC takes over.
>>
>
> The fact that you start seeing the concurrent mode failures after
> 24 hours indicates to me strongly that the old generation gets slowly
> fragmented over a period of time. (Recall that the CMS collector is
> non-moving.)
>
> Can you confirm that the heap occupancy itself is constant (or nearly
> so) following CMS collection cycles, and that the full gc that follows
> a concurrent mode failure does not unload classes? Recall that CMS will
> not, by default, unload classes during concurrent cycles unless
> explicitly instructed to do so via:
>
>      -XX:+CMSClassUnloadingEnabled -XX:+PermGenSweepingEnabled
>
> (the second option is needed in pre-6.0 JVM's, but not in more
> recent JVM's).
>
>> I have tried varying the CMSInitiatingOccupancyFraction to 20%, but 
>> the CMS mode failures still occur.
>
> It is usually a good idea to use survivor spaces to both reduce the
> pressure on the concurrent collector (by promoting less to the old
> gen), but also to reduce the spread in object sizes and lifetimes
> of the objects that do get promoted. I'd suggest using survivor
> spaces to make sure that survivors stay in the young gen for at least
> one scavenge (MaxTenuringThreshold = 1, possibly more, as experiments
> dictate), possibly more. A downside is possibly longer scavenges,
> but consider that the price for (possibly) avoiding concurrent
> mode failure.
>
> Prematurely promoting objects (besides the two points made above),
> can also reduce floating garbage and reduce CMS remark pause
> times (by reducing mutation rates in the old generation).
>
>>
>> I am now running with the incremental mode CMS; but anticipate 
>> further very long pauses.
>
> From what you described above (running CMS all the time by setting the
> initiation threshold very low), it does not look as though iCMS will
> buy you anything.
>
>>
>> The VM always recovers very well after these sporadic Full GCs, but 
>> to eradicate them, should I run with an 8 GB heap or something along 
>> those lines.? I also read something about killing the swap file?
>>
>> My AMD 64 bit bx, unfortunately for now is restricted to 4 GB RAM; 
>> but I am adding a further 4 GB soon. I am about to go to the Solaris 
>> SPARC 64 bit and run the exact same scenario with a 7-8 GB heap.
>
> Increasing the heap size can indeed sometimes help you avoid
> concurrent mode failure from fragmentation. (But first make
> sure to enable survivor spaces and, if applicable, perm gen
> collection.)
>
>>
>> I read about the occupancy fration for OG and Perm Gen; do I need to 
>> apply this patch. Our Perm Gen is always set to 128 MB and only ever 
>> attains 108 MB.
>
> The webrev i posted late last week should not really apply directly to
> your case (except inasmuch as, in the event that you enable perm gen
> collection, it might allow you to get away with not collecting the
> perm gen per each cycle, and thus help keep cms remark pauses possibly
> shorter). I would not worry about this patch at the level at which you
> are tuning currently (which is mainly looking to avoid the concurrent
> mode failures).
>
> -- ramki
>
>>
>> Any feedback would help us in our endeavours to support our EBI apps 
>> in a 64 bit env.
>>
>> keith
>>
>>
>> Keith R Holdaway
>> Java Development Technologies
>>
>> SAS...  The Power to Know
>>
>> Carpe Diem ...
>>
>

-- 
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)



From Keith.Holdaway at sas.com  Mon Dec 10 11:55:49 2007
From: Keith.Holdaway at sas.com (Keith Holdaway)
Date: Mon, 10 Dec 2007 14:55:49 -0500
Subject: 64 bit CMS JDK 5.0 u14
In-Reply-To: <475D8C75.5010509@sun.com>
References: <fc8ea97d6b9c.47597598@sun.com>
	<304E9E55F6A4BE4B910C2437D4D1B49608FB48C825@MERCMBX14.na.sas.com>
	<475D8920.8000501@Sun.COM> <475D8C75.5010509@sun.com>
Message-ID: <304E9E55F6A4BE4B910C2437D4D1B49608FB48CB9D@MERCMBX14.na.sas.com>

Tony,

I intend to elaborate/add GC logs later; but to quickly dispell any misunderstanding. We currently deploy most of our middle-tier apps running in a VM implementing CMS; the one major EBI app that tends not to play well with CMS is our Portal app. However, at 64-bit, the Portal performs as well as it does in 32 bit with the throughput collector.

I prefer the CMS algorithm since the low pause is crucial. The Portal generates a great deal of small short-lived objects; and sometimes in 32 bit mode we overrun the algorithm, even using survivor spaces; fragmentation is usually an issue with our CMS experieces in 32 bit with Portal. I, too, am very happy with CMS at 32-bit; just a little hard to fine tune our Portal with it; but again I aware of small memory leaks with Portal.

I am very encouraged by the CMS algorithm on AMD 64 bit with Portal; just a little concerned about some of the 400-600 seconds Full GCs when the concurrent mode failures kick in. Fragmentation is at play for sure; so utilizing the SS1 and SS2  may help. The promotion guarantee is still an issue at 5.0 u14?

I shall add more details later. But I really appreciate all your feedback.

keith


Keith R Holdaway
Java Development Technologies

SAS...  The Power to Know

Carpe Diem ...

-----Original Message-----
From: Tony Printezis [mailto:tony.printezis at sun.com]
Sent: Monday, December 10, 2007 1:59 PM
To: Y.S.Ramakrishna at sun.com
Cc: Keith Holdaway; hotspot-gc-dev at openjdk.java.net
Subject: Re: 64 bit CMS JDK 5.0 u14

Keith and Ramki,

I'd like to add a couple of things to Ramki's informative reply.

First, I wonder why the Portal ran fine with the default GC in 1.4.2
with no failed (I assumed timed out, right?) transactions. The default
(aka serial) GC is not a low-pause GC, so maybe the pause times it gave
you were good enough. BTW, what are your latency targets? hundreds of
ms? seconds?

Second, we'd be able to help you a bit more constructively if you gave
us some GC logs to look at (and if you have both WLS 8.1 and 9.2 logs,
it'd be nice to get them so that we can compare them).

 > CMS never works very well in the 32-bit environment

FWIW, a lot of our customers are happy with CMS in a 32-bit environment! :-)

Tony

Y.S.Ramakrishna at sun.com wrote:
> Hi Keith --
>
>> I am running some midle-tier Portal load tests in WLS 9.2 MP2 with
>> Sun JDK 5.0 u14. I am running 100 concurrent users that logon,
>> navigate, open portlets, and eventually logoff; only to logon again
>> and repeat the cycle. My testing people have establish Load Runner
>> scripts to put the Portal software through an endurance test over 5
>> days with the 100 users.
>>
>> Normally on 32-bit WLS 8.1 SP6 with JDK 1.4.2_13 we run with the
>> following VM args; we attain some 3.4 million passed transactions
>> with zero failed transactions:
>>
>> -server -Xms1400m -Xmx1400m -XX:NewSize=64m -XX:MaxNewSize=64m
>> -XX:PermSize=128m -XX:MaxPermSize=128m -Xss128k -XX:-UseTLAB
>> -XX:+DisableExplicitGC
>> -Dsun.rmi.dgc.client.gcInterval=3600000
>> -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true
>>
>> CMS never works very well in the 32-bit environment; failing
>> miserably above; although at JDK 1.4.2_15, we see some 2 million
>> passed transactions with 1130+ failed transactions owing to 120
>> seconds timeouts in concurrent mode failures.
>>
>> Now, in the 64 bit environment running on AMD Windows Server 2003, I
>> can run pretty successfully with CMS:
>>
>> -Xms1500m -Xmx3500m -XX:NewSize=320m -XX:MaxNewSize=320m -Xss256k
>> -XX:PermSize=128m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC
>> -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly
>> -XX:CMSInitiatingOccupancyFraction=40
>> -Dsun.rmi.dgc.client.gcInterval=3600000
>> -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true
>> -Dcom.sun.management.jmxremote -verbosegc
>> -Xloggc:C:\keith\GCLogs\gc11.txt
>>
>> I can achieve some 3.4 million as for the throughput collector in 32
>> bit env.
>>
>> But, I do note several thousand failed transactions that correlate
>> with concurrent mode failures after some 24 hrs; pauses in the
>> 400-600 seconds range when Full GC takes over.
>>
>
> The fact that you start seeing the concurrent mode failures after
> 24 hours indicates to me strongly that the old generation gets slowly
> fragmented over a period of time. (Recall that the CMS collector is
> non-moving.)
>
> Can you confirm that the heap occupancy itself is constant (or nearly
> so) following CMS collection cycles, and that the full gc that follows
> a concurrent mode failure does not unload classes? Recall that CMS will
> not, by default, unload classes during concurrent cycles unless
> explicitly instructed to do so via:
>
>      -XX:+CMSClassUnloadingEnabled -XX:+PermGenSweepingEnabled
>
> (the second option is needed in pre-6.0 JVM's, but not in more
> recent JVM's).
>
>> I have tried varying the CMSInitiatingOccupancyFraction to 20%, but
>> the CMS mode failures still occur.
>
> It is usually a good idea to use survivor spaces to both reduce the
> pressure on the concurrent collector (by promoting less to the old
> gen), but also to reduce the spread in object sizes and lifetimes
> of the objects that do get promoted. I'd suggest using survivor
> spaces to make sure that survivors stay in the young gen for at least
> one scavenge (MaxTenuringThreshold = 1, possibly more, as experiments
> dictate), possibly more. A downside is possibly longer scavenges,
> but consider that the price for (possibly) avoiding concurrent
> mode failure.
>
> Prematurely promoting objects (besides the two points made above),
> can also reduce floating garbage and reduce CMS remark pause
> times (by reducing mutation rates in the old generation).
>
>>
>> I am now running with the incremental mode CMS; but anticipate
>> further very long pauses.
>
> From what you described above (running CMS all the time by setting the
> initiation threshold very low), it does not look as though iCMS will
> buy you anything.
>
>>
>> The VM always recovers very well after these sporadic Full GCs, but
>> to eradicate them, should I run with an 8 GB heap or something along
>> those lines.? I also read something about killing the swap file?
>>
>> My AMD 64 bit bx, unfortunately for now is restricted to 4 GB RAM;
>> but I am adding a further 4 GB soon. I am about to go to the Solaris
>> SPARC 64 bit and run the exact same scenario with a 7-8 GB heap.
>
> Increasing the heap size can indeed sometimes help you avoid
> concurrent mode failure from fragmentation. (But first make
> sure to enable survivor spaces and, if applicable, perm gen
> collection.)
>
>>
>> I read about the occupancy fration for OG and Perm Gen; do I need to
>> apply this patch. Our Perm Gen is always set to 128 MB and only ever
>> attains 108 MB.
>
> The webrev i posted late last week should not really apply directly to
> your case (except inasmuch as, in the event that you enable perm gen
> collection, it might allow you to get away with not collecting the
> perm gen per each cycle, and thus help keep cms remark pauses possibly
> shorter). I would not worry about this patch at the level at which you
> are tuning currently (which is mainly looking to avoid the concurrent
> mode failures).
>
> -- ramki
>
>>
>> Any feedback would help us in our endeavours to support our EBI apps
>> in a 64 bit env.
>>
>> keith
>>
>>
>> Keith R Holdaway
>> Java Development Technologies
>>
>> SAS...  The Power to Know
>>
>> Carpe Diem ...
>>
>

--
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)



From Y.S.Ramakrishna at Sun.COM  Mon Dec 17 16:00:18 2007
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Mon, 17 Dec 2007 16:00:18 -0800
Subject: equest for review (small): 6621728
	"Heap inspection should not crash in the face of C-heap exhaustion"
Message-ID: <f821b2813ea0.47669d12@sun.com>


[A set of patches as well as a gzipped tarball of the webrev is attached.
Should your mail servers strip either of the attachments, you may manually
access them from the archives of this mailing list. For those internal
to Sun, a webrev pointer of these changes can be found off of
the "Suggested Fix" section of the CR. Thanks a lot for your reviews!]
-----------------------------

Fixed 6621728: Heap inspection should not crash in the face of C-heap exhaustion

When requisite support data structures for accumulating the statistics
cannot be allocated because of C-heap exhaustion, we now either simply
refuse to compute the histogram information or report the extent of
undercounting (depending on which allocation(s) did not succeed).

[*] I also took the opportunity to clean up and consolidate some
statistics reporting associated with CMS's free list spaces that
came to light from a recent customer interaction.

The changes directly related to the original bug are the first two
files in the webrev (or in the patch file). The remaining files contain
changes related to the clean-up in [*] above.

Fix verified: yes

Verification Testing: artificially injected allocation
    failures; tested +PrintClassHistogram with SPECjvm98;
    dumped FLS census data and checked formatting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cms_bugs.patch
Type: text/x-patch
Size: 38849 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20071217/a87eeb98/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: webrev.6621728.tar.gz
Type: application/x-gzip
Size: 2125391 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20071217/a87eeb98/attachment-0003.bin 

From Keith.Holdaway at sas.com  Thu Dec 20 08:16:28 2007
From: Keith.Holdaway at sas.com (Keith Holdaway)
Date: Thu, 20 Dec 2007 11:16:28 -0500
Subject: HOARD
Message-ID: <304E9E55F6A4BE4B910C2437D4D1B49608FC4CC893@MERCMBX14.na.sas.com>

I researched HOARD:

http://developers.sun.com/solaris/articles/multiproc/multiproc.html

The Hoard memory allocator is a fast, scalable, and memory-efficient memory allocator. It runs on a variety of platforms, including Linux, Solaris, and Windows. Hoard is a drop-in replacement for malloc() that can dramatically improve application performance, especially for multithreaded programs running on multiprocessors.

Is this purely an OS change; or does one have to make changes to the VM source to use HOARD instead of malloc etc?

keith


Keith R Holdaway
Java Development Technologies<http://rdwss.na.sas.com/mti/jdt/default.aspx>

SAS...  The Power to Know

Carpe Diem ...



From tony.printezis at sun.com  Thu Dec 20 08:21:47 2007
From: tony.printezis at sun.com (Tony Printezis)
Date: Thu, 20 Dec 2007 11:21:47 -0500
Subject: HOARD
In-Reply-To: <304E9E55F6A4BE4B910C2437D4D1B49608FC4CC893@MERCMBX14.na.sas.com>
References: <304E9E55F6A4BE4B910C2437D4D1B49608FC4CC893@MERCMBX14.na.sas.com>
Message-ID: <476A969B.6020803@sun.com>

Keith,

Hi. You should just be able to use Hoard instead of whatever malloc 
comes with the OS without and OS changes.

I just don't know whether the JVM will actually benefit from Hoard 
though. We don't do that much malloc within the JVM (we manage all Java 
objects ourselves). It'd be a nice exercise, though, if you're willing 
to try it and share your experience on the list! :-) Maybe, it could be 
a win if you have a native library that calls malloc extensively and 
from many threads.

I'm CCing Mr Hoard so that he can share his wisdom. :-)

Regards and happy holidays everyone!

Tony

Keith Holdaway wrote:
> I researched HOARD:
>
> http://developers.sun.com/solaris/articles/multiproc/multiproc.html
>
> The Hoard memory allocator is a fast, scalable, and memory-efficient memory allocator. It runs on a variety of platforms, including Linux, Solaris, and Windows. Hoard is a drop-in replacement for malloc() that can dramatically improve application performance, especially for multithreaded programs running on multiprocessors.
>
> Is this purely an OS change; or does one have to make changes to the VM source to use HOARD instead of malloc etc?
>
> keith
>
>
> Keith R Holdaway
> Java Development Technologies<http://rdwss.na.sas.com/mti/jdt/default.aspx>
>
> SAS...  The Power to Know
>
> Carpe Diem ...
>
>
>   

-- 
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)



From Y.S.Ramakrishna at Sun.COM  Thu Dec 20 10:14:18 2007
From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna)
Date: Thu, 20 Dec 2007 10:14:18 -0800
Subject: HOARD
In-Reply-To: <304E9E55F6A4BE4B910C2437D4D1B49608FC4CC893@MERCMBX14.na.sas.com>
References: <304E9E55F6A4BE4B910C2437D4D1B49608FC4CC893@MERCMBX14.na.sas.com>
Message-ID: <f7e9d79b65ec.476a407a@sun.com>


Hi Keith --

> I researched HOARD:
> 
> http://developers.sun.com/solaris/articles/multiproc/multiproc.html
> 
> The Hoard memory allocator is a fast, scalable, and memory-efficient 
> memory allocator. It runs on a variety of platforms, including Linux, 
> Solaris, and Windows. Hoard is a drop-in replacement for malloc() that 
> can dramatically improve application performance, especially for 
> multithreaded programs running on multiprocessors.
> 
> Is this purely an OS change; or does one have to make changes to the 
> VM source to use HOARD instead of malloc etc?

Have you found that malloc() becomes a bottleneck in your application?
The JVM's use of the C-heap should usually not be more than a minor
blip on the performance profile, as such i would not think that Hoard's
scalability and performance would contribute much to the bottom line
of JVM performance, at least wrt pure Java applications.

On the other hand, if your application has a lot of native (JNI) code that
relies on the use of the C-heap (especially in a multi-threaded way)
then it's likely that your application will benefit from the use of Hoard.

As pointed out by the authors in the article, it should suffice (at least
on Solaris) to have your environment with LD_PRELOAD defined to include
the location of libhoard.so. My guess would be that the VM would then
use libhoard.so's malloc in preference to libc.so's malloc.

However, i have never personally tried this myself. I think members of the
runtime team have, in the past, experimented with alternate
malloc implementations, perhaps including Hoard. I am cross-posting
to their mailing list and hopefully they will be able to provide more
useful advice & help on this.

-- ramki

> 
> keith
> 
> 
> Keith R Holdaway
> Java Development Technologies<
> 
> SAS...  The Power to Know
> 
> Carpe Diem ...
> 
> 

From emery at cs.umass.edu  Thu Dec 20 10:36:13 2007
From: emery at cs.umass.edu (Emery Berger)
Date: Thu, 20 Dec 2007 13:36:13 -0500
Subject: HOARD
References: <304E9E55F6A4BE4B910C2437D4D1B49608FC4CC893@MERCMBX14.na.sas.com>
	<476A969B.6020803@sun.com>
Message-ID: <2330F00E4A3C924AB7F7297AE42D35351CF6DF@zor.ads.cs.umass.edu>

Hi all,
 
I know that Novell has used Hoard to improve performance of its VM, but in that VM, Java objects are allocated from the C heap. Given Tony's comments, I would not expect Hoard to make any difference for pure Java code running in HotSpot, but could certainly impact performance for native libraries. I'd be interested to hear the results.
 
Best,
-- Emery (apparently a.k.a. "Mr. Hoard" :))
 
---
Emery Berger
Assistant Professor
Dept. of Computer Science
University of Massachusetts Amherst
http://www.cs.umass.edu/~emery <http://www.cs.umass.edu/~emery> 


________________________________

From: Tony Printezis [mailto:tony.printezis at sun.com]
Sent: Thu 12/20/2007 11:21 AM
To: Keith Holdaway
Cc: hotspot-gc-dev at openjdk.java.net; Emery Berger
Subject: Re: HOARD



Keith,

Hi. You should just be able to use Hoard instead of whatever malloc
comes with the OS without and OS changes.

I just don't know whether the JVM will actually benefit from Hoard
though. We don't do that much malloc within the JVM (we manage all Java
objects ourselves). It'd be a nice exercise, though, if you're willing
to try it and share your experience on the list! :-) Maybe, it could be
a win if you have a native library that calls malloc extensively and
from many threads.

I'm CCing Mr Hoard so that he can share his wisdom. :-)

Regards and happy holidays everyone!

Tony

Keith Holdaway wrote:
> I researched HOARD:
>
> http://developers.sun.com/solaris/articles/multiproc/multiproc.html
>
> The Hoard memory allocator is a fast, scalable, and memory-efficient memory allocator. It runs on a variety of platforms, including Linux, Solaris, and Windows. Hoard is a drop-in replacement for malloc() that can dramatically improve application performance, especially for multithreaded programs running on multiprocessors.
>
> Is this purely an OS change; or does one have to make changes to the VM source to use HOARD instead of malloc etc?
>
> keith
>
>
> Keith R Holdaway
> Java Development Technologies<http://rdwss.na.sas.com/mti/jdt/default.aspx>
>
> SAS...  The Power to Know
>
> Carpe Diem ...
>
>
>  

--
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20071220/98b6d979/attachment.html 

From Dave.Dice at Sun.COM  Thu Dec 20 10:51:16 2007
From: Dave.Dice at Sun.COM (Dave Dice)
Date: Thu, 20 Dec 2007 13:51:16 -0500
Subject: HOARD
In-Reply-To: <f7e9d79b65ec.476a407a@sun.com>
Message-ID: <0JTD0008W2YCJK40@mail-amer.sun.com>

Hi Ramki,

Hoard is perfectly good allocator, and as you noted, you can use LD_PRELOAD to
enable it.  (IIRC, hoard.org provides SPARC .so-es for download, so it's nearly
painless to try).   At various times I've tried it with hotspot and it behaved
well.  The latest version uses per-thread subheaps, so there's no
synchronization (contended or uncontended) for "normal" allocation and
deallocation.  Normal being defined as the case where the same thread that
allocates a block also frees the block.  If on Solaris, however, I'd recommend
libumem in slight preference over hoard.  Libumem is fully supported and tuned
specifically for solaris.   Libumem and hoard share many concepts, the key being
the use of subheaps to diffuse contention and to avoid false sharing.  They both
suffer from increased footprint, however, at least as compared to the default
allocators.  The Lea allocator (the default on linux) and the solaris libc
allocator (based on Tarjan trees augmented with segregated free lists for common
small sizes) use a common heap and a global local.  They both have excellent
footprint and heap density.  

Beyond footprint, the only problem I've ever run into with hoard or libumem was
a red-black tree where various threads were adding and removing nodes on MP
systems.  Over time the traversal from the root to the leaves ended up touching
nodes allocated by different threads, and thus in different subheaps, and thus
covered by different (small page) TLBs.  (You get the picture (:>)).  

Regards
Dave


p.s., really old JVMs use the __malloc_lock around thread suspension to avoid
deadlock, in which case you can't substitute allocators. 



| -----Original Message-----
| From: hotspot-runtime-dev-bounces at openjdk.java.net 
| [mailto:hotspot-runtime-dev-bounces at openjdk.java.net] On 
| Behalf Of Y Srinivas Ramakrishna
| Sent: Thursday, December 20, 2007 1:14 PM
| To: Keith Holdaway
| Cc: hotspot-gc-dev at openjdk.java.net; 
| hotspot-runtime-dev at openjdk.java.net
| Subject: Re: HOARD
| 
| 
| Hi Keith --
| 
| > I researched HOARD:
| > 
| > http://developers.sun.com/solaris/articles/multiproc/multiproc.html
| > 
| > The Hoard memory allocator is a fast, scalable, and 
| memory-efficient 
| > memory allocator. It runs on a variety of platforms, 
| including Linux, 
| > Solaris, and Windows. Hoard is a drop-in replacement for 
| malloc() that 
| > can dramatically improve application performance, especially for 
| > multithreaded programs running on multiprocessors.
| > 
| > Is this purely an OS change; or does one have to make 
| changes to the 
| > VM source to use HOARD instead of malloc etc?
| 
| Have you found that malloc() becomes a bottleneck in your application?
| The JVM's use of the C-heap should usually not be more than a 
| minor blip on the performance profile, as such i would not 
| think that Hoard's scalability and performance would 
| contribute much to the bottom line of JVM performance, at 
| least wrt pure Java applications.
| 
| On the other hand, if your application has a lot of native 
| (JNI) code that relies on the use of the C-heap (especially 
| in a multi-threaded way) then it's likely that your 
| application will benefit from the use of Hoard.
| 
| As pointed out by the authors in the article, it should 
| suffice (at least on Solaris) to have your environment with 
| LD_PRELOAD defined to include the location of libhoard.so. My 
| guess would be that the VM would then use libhoard.so's 
| malloc in preference to libc.so's malloc.
| 
| However, i have never personally tried this myself. I think 
| members of the runtime team have, in the past, experimented 
| with alternate malloc implementations, perhaps including 
| Hoard. I am cross-posting to their mailing list and hopefully 
| they will be able to provide more useful advice & help on this.
| 
| -- ramki
| 
| > 
| > keith
| > 
| > 
| > Keith R Holdaway
| > Java Development Technologies<
| > 
| > SAS...  The Power to Know
| > 
| > Carpe Diem ...
| > 
| > 
| 


From jason at mugfu.com  Mon Dec 31 10:21:08 2007
From: jason at mugfu.com (Jason Vasquez)
Date: Mon, 31 Dec 2007 13:21:08 -0500
Subject: Perplexing GC Time Growth
Message-ID: <C1645320-A2CC-4E58-9DF9-93E657976E68@mugfu.com>

Hi all,

I'm having a perplexing problem -- the garbage collector appears to be  
functioning well, with a nice object/garbage lifecycle, yet minor GC  
times increase over the life of the process inexplicably.  We are  
working with telephony hardware with this application, so keeping GC  
pauses very low is quite important.  (keeping well below 100 ms would  
be ideal)

Here is the current configuration we are using:

-server \
-Xloggc:garbage.log \
-XX:+PrintGCDetails \
-Dsun.rmi.dgc.server.gcInterval=3600000 \
-Dsun.rmi.dgc.client.gcInterval=3600000 \
-XX:ParallelGCThreads=8 \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+PrintGCTimeStamps \
-XX:-TraceClassUnloading \
-XX:+AggressiveOpts \
-Xmx512M \
-Xms512M \
-Xmn128M \
-XX:MaxTenuringThreshold=6 \
-XX:+ExplicitGCInvokesConcurrent

A large number of our bigger objects size-wise live for approximately  
4-5 minutes, thus the larger young generation, and tenuring threshold.  
This seems to be successful in filtering most objects before they  
reach the tenured gen. (8 core x86 server, running 1.6.0_03-b05 on 32- 
bit Linux, kernel rev 2.6.18)

Here is a representative snippet of our garbage log:

487.135: [GC 487.135: [ParNew: 112726K->7290K(118016K), 0.0218110  
secs] 134494K->29058K(511232K), 0.0220520 secs]
557.294: [GC 557.294: [ParNew: 112250K->7976K(118016K), 0.0204220  
secs] 134018K->29744K(511232K), 0.0206690 secs]
607.025: [GC 607.025: [ParNew: 112936K->7831K(118016K), 0.0231230  
secs] 134704K->30003K(511232K), 0.0233670 secs]
672.522: [GC 672.522: [ParNew: 112791K->7361K(118016K), 0.0253620  
secs] 134963K->29533K(511232K), 0.0256080 secs]
...
4006.635: [GC 4006.635: [ParNew: 112983K->7386K(118016K), 0.0385960  
secs] 141969K->36608K(511232K), 0.0388460 secs]
4083.066: [GC 4083.066: [ParNew: 112346K->8439K(118016K), 0.0365940  
secs] 141568K->37661K(511232K), 0.0368340 secs]
4158.457: [GC 4158.457: [ParNew: 113399K->7152K(118016K), 0.0360540  
secs] 142621K->36374K(511232K), 0.0362990 secs]
4228.312: [GC 4228.313: [ParNew: 112112K->8738K(118016K), 0.0362510  
secs] 141334K->38083K(511232K), 0.0365050 secs]
4293.800: [GC 4293.800: [ParNew: 113698K->8294K(118016K), 0.0368700  
secs] 143043K->37917K(511232K), 0.0371160 secs]
...
10489.555: [GC 10489.556: [ParNew: 112701K->7770K(118016K), 0.0639770  
secs] 151966K->47156K(511232K), 0.0642210 secs]
10562.544: [GC 10562.544: [ParNew: 112730K->9267K(118016K), 0.0625900  
secs] 152116K->48772K(511232K), 0.0628470 secs]
10622.558: [GC 10622.558: [ParNew: 114227K->8361K(118016K), 0.0675730  
secs] 153732K->48381K(511232K), 0.0678220 secs]
10678.842: [GC 10678.842: [ParNew: 113056K->7214K(118016K), 0.0669330  
secs] 153076K->47234K(511232K), 0.0671800 secs]
...
177939.062: [GC 177939.062: [ParNew: 112608K->8620K(118016K),  
0.7681440 secs] 466132K->362144K(511232K), 0.7684030 secs]
178005.483: [GC 178005.483: [ParNew: 113449K->7731K(118016K),  
0.7677300 secs] 466973K->361893K(511232K), 0.7679890 secs]
178069.658: [GC 178069.658: [ParNew: 112670K->6814K(118016K),  
0.7700020 secs] 466832K->360976K(511232K), 0.7702590 secs]
178133.513: [GC 178133.513: [ParNew: 111717K->7920K(118016K),  
0.7702920 secs] 465879K->362082K(511232K), 0.7705560 secs]

As you can see, the gc times continue to increase over time, on the  
order of about 10-20ms per hour.  CMS runs are spaced very far apart,  
in fact, since most objects die before reaching the tenured  
generation, the CMS is triggered more by RMI DGC runs then by heap  
growth.   (We were getting serial GCs, apparently due to RMI DGC  
before adding -XX:+ExplicitGCInvokesConcurrent)

Here's some representative output from `jstat -gcutil -t -h10 <pid> 2s`:

Timestamp S0 S1 E O P YGC YGCT FGC FGCT GCT
11067.6 55.74 0.00 89.32 9.73 59.84 168 7.471 8 0.280 7.751
11069.6 55.74 0.00 93.65 9.73 59.84 168 7.471 8 0.280 7.751
11071.6 55.74 0.00 99.34 9.73 59.84 168 7.471 8 0.280 7.751
11073.5 0.00 62.22 2.89 9.76 59.84 169 7.537 8 0.280 7.816
11075.6 0.00 62.22 4.80 9.76 59.84 169 7.537 8 0.280 7.816

Survivor spaces continue to sit at about 50-65% occupancy, which seems  
fairly good to my eye.  Eden fills approximately every 70 seconds,  
triggering minor GCs.


Any ideas?  This is becoming quite frustrating for us -- our  
application uptime is pretty horrible with the too-frequent scheduled  
restarts we are being forced to run.

Thanks for any assistance you might be able to offer,
-Jason Vasquez
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use