Fwd: CMS Garbage collection eating up processor power

Michael Finocchiaro michael.finocchiaro at gmail.com
Mon Mar 31 09:17:38 UTC 2008


Hi again,

  I think that the Eden should be increased to filter more objects from
  getting promoted to Old.
  Could -XX:SoftRefLRUPolicyMSPerMB=1 help here? Perhaps there are lots
  of soft references that are clogging the heap?
  Another possibility could be finalzers. Are there lots of finalizers>
  in the Portal Server code? If you have JMX deployed in the server,
  you can check the outstanding finalized objects via jConsole.
  Otherwise, there are some little known options:
  -Djava.finalizer.verbose=[true | false]
  -Djava.finalizer.verbose.rate=[frequency of logging in seconds]
  -Djava.finalizer.verbose.output=[filename]
  -D.java.finalizer.threadCount=[# of finalizer threads]

  The output looks like this:

  <F: version = 1.4.2.10 <http://1.4.2.10> >
  <F: java.finalizer.threadCount = 1 >
  <F: 1 54982 123064 147 >
  <F: 2 170644 166824 284 >
  <F: 3 251356 172390 94260 >
  <F: 4 344950 187071 203 >

  <F: a b c d >
  a = sample number
  b = milliseconds since start
  c = count of live and pending objects with finalizers
  d = count of pending objects with finalizers
  If the number of pending finalizers continues to increase without
  bound or the number of pending
  objects is consistently over 100000, you can increase the number of
  threads available for finalizing
  objects to 2 with the following option:

  -Djava.finalizer.threadCount=2

  Note that you should keep the finalizer.threadCount much lower than
  the number of CPUs on the machine – like in the previous discussion of
  ParallelGCThreads.

  I have encountered this issue on older version of Tomcat and this
  helped reduce the problem.

  Hope this helps,
  Fino
>
> On Fri, Mar 28, 2008 at 8:39 AM, Jon Masamitsu <Jon.Masamitsu at sun.com
> <mailto:Jon.Masamitsu at sun.com>> wrote:
>
>     T.K wrote On 03/27/08 20:19,:
>
>     > I've attached an old gclog where it has similar behavior on the
>     heap.
>     > But the problem with that is it takes quite a long time, like 20
>     - 70
>     > seconds.  The server basically went down with customers complaining
>     > during that time, so we had to try CMS, but that doesn't work
>     out well
>     > either.
>
>
>     A significant fraction of your heap is filled with live data.  The
>     GC is
>     running
>     almost constantly because it is trying to keep the application
>     going by
>     recovering
>     whatever space it can.  If the occupancy of the heap does not go
>     down when
>     the number of users drops, there's some code in the application
>     that is
>     keeping objects alive that perhaps are never going to be used
>     again.  If the
>     application really needs all that data, then you need a larger
>     heap to keep
>     GC from running so frequently.   If increasing the size of the
>     heap just
>     delays the onset of the frequent GC's and you do think that all
>     that data
>     should be live, you may have to limit the number of users in some
>     way so
>     as to not fill up the heap.
>
>     >
>     > We are having Sun Portal Server 6.2 on those web servers.  Do
>     you know
>     > is it a normal behavior for Portal Server with about 400-500
>     users per
>     > instance?
>
>
>     Don't know about this.
>
>     > Thanks,
>     > TK
>     >
>     > On Thu, Mar 27, 2008 at 5:10 PM, Jon Masamitsu
>     <Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>
>     > <mailto:Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>>>
>     wrote:
>     >
>     >     Late in this log I see
>     >
>     >     162417.510: [GC 162417.510: [ParNew: 370176K->11520K(381696K),
>     >     0.4232456
>     >     secs] 3055096K->2715938K(3134208K), 0.4240171 secs]
>     >
>     >     At that point the heap is about 85% full
>     (2715938K/3134208K).  The
>     >     tenured generation is almost completely full.
>     >
>     >     Do you have similar logs when using the default GC?  We
>     could use them
>     >     to verify the amount of live data.
>     >
>     >
>     >     T.K wrote:
>     >     >
>     >     > Hi Jon,
>     >     >
>     >     > Here's the attached gclog.  It starts out fine when we put in
>     >     the change
>     >     > in the evening.  Eventually, the next morning (around 60000
>     >     seconds on
>     >     > gc time) when the load starting to get in, CMS starts to run
>     >     > consectively.  The first CMS fails occur when we bump up the
>     >     users to
>     >     > 700+, and almost kill the server.  Ever since then, I
>     don't see
>     >     the CMS
>     >     > ever stops, even when the load goes down to 50 users.
>     >     >
>     >     > I cut off the logs in between into 3 portion so that I can
>     >     attach it.  :D
>     >     >
>     >     >
>     >     > Thanks,
>     >     > TK
>     >     >
>     >     >
>     >     > On 3/27/08, *Jon Masamitsu* <Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com>
>     >     <mailto:Jon.Masamitsu at sun.com <mailto:Jon.Masamitsu at sun.com>>
>     >     > <mailto:Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com> <mailto:Jon.Masamitsu at sun.com
>     <mailto:Jon.Masamitsu at sun.com>>>>
>     >     wrote:
>     >     >
>     >     >     Using CMS sometimes needs some tuning (especially
>     >     >     with the 1.4.2 jdk).  Do you have any gc logs
>     >     >     (-XX:+PrintGCDetails) so we can see what's happening?
>     >     >
>     >     >
>     >     >     T.K wrote:
>     >     >      > Hi All,
>     >     >      > We got 5 Sun Web Servers running on Java 1.4.2, and
>     used
>     >     to use the
>     >     >      > default GC for Tenured space. The problem with that is
>     >     that it takes
>     >     >      > 60-80 seconds everytime the GC happens, and the latency
>     >     on the
>     >     >     site goes
>     >     >      > crazy. So we decided to change it to use the Concurrent
>     >     Mark Sweep
>     >     >      > Collector on one server to test it out. Here's the
>     setting:
>     >     >      >
>     >     >      > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xms3G -Xmx3G
>     >     >     -XX:NewSize=384M
>     >     >      > -XX:MaxNewSize=384M -XX:PermSize=64M
>     -XX:MaxPermSize=64M
>     >     >      > -XX:CMSInitiatingOccupancyFraction=60
>     >     >      >
>     >     >      > With that setting, the server runs great. But
>     eventually,
>     >     when the
>     >     >      > server reach a medium load (around 100-200 users), the
>     >     tenured
>     >     >     space is
>     >     >      > always around half full, and the CMS collector
>     starts to run
>     >     >      > continuously one after another. It doesn't hurt the
>     >     application
>     >     >     for now,
>     >     >      > but it's taking 25% of processing time (we got 4
>     cpu, so
>     >     one web
>     >     >     server
>     >     >      > always keep 1 cpu power). I don't see that much cpu
>     >     utilization
>     >     >     on other
>     >     >      > web server that don't have CMS, and they have more
>     users
>     >     than the one
>     >     >      > with CMS. If we got CMS on all 5 web servers, I'm
>     >     wondering if
>     >     >     that will
>     >     >      > crash the server or not.  What should I do to decrease
>     >     the processor
>     >     >      > utilization caused by GC?
>     >     >      >
>     >     >      > Also, I'm thinking to use i-CMS on the JVM, and maybe
>     >     that might slow
>     >     >      > down the CMS and reduce the amount of CPU
>     utilization by
>     >     CMS. Any
>     >     >     thought?
>     >     >      >
>     >     >      > Thanks,
>     >     >      > TK
>     >     >      >
>     >     >      >
>     >     >      >
>     >     >
>     >
>
------------------------------------------------------------------------
>     >     >      >
>     >     >      > _______________________________________________
>     >     >      > hotspot-gc-use mailing list
>     >     >      > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>
>     >     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>>
>     >     >      >
>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>     >     >
>     >     >
>     >     >
>     >     >
>     >
>
------------------------------------------------------------------------
>     >     >
>     >     > _______________________________________________
>     >     > hotspot-gc-use mailing list
>     >     > hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     >     <mailto:hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>>
>     >     > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>     >
>     >
>
>------------------------------------------------------------------------
>     >
>     >
>     >
>     >
>
>     _______________________________________________
>     hotspot-gc-use mailing list
>     hotspot-gc-use at openjdk.java.net
>     <mailto:hotspot-gc-use at openjdk.java.net>
>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>
>
> --
> Michael Finocchiaro
> michael.finocchiaro at gmail.com <mailto:michael.finocchiaro at gmail.com>
> Mobile Telephone: +33 6 67 90 64 39
> MSN: le_fino at hotmail.com <mailto:le_fino at hotmail.com>






-- 
Michael Finocchiaro
michael.finocchiaro at gmail.com
Mobile Telephone: +33 6 67 90 64 39
MSN: le_fino at hotmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20080331/fcd96c8b/attachment.htm>
-------------- next part --------------
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


More information about the hotspot-gc-dev mailing list