the amazing tales of the search for the invisible man! or, where's my gc root
jed at atlassian.com
Fri Apr 17 16:32:25 PDT 2009
Thanks for the info. I will try NetBeans and look at what it tells me
- this is certainly a lot more useful than what the other tools have
been able to give me. Unfortunately it is Saturday morning in Sydney
and I will not be able to attempt with generations on until Monday.
As reported below, we have verified that the IBM JVM does not leak the
plugin system classes.
On 18/04/2009, at 3:51 AM, kirk wrote:
> Hi Jed,
> I've had a quick look at the heap dump. I'm having a little trouble
> understanding what is in there. What I can see is a large number of
> java.lang.reflect.Method objects being held. There seems to be two
> competing patterns of references holding onto these objects. I've
> attached some screenshots rather than use words.
> The scary thing is that the references include ClassLoader.scl,
> JDK12Hooks.systemClassLoader as well as Apache commons logging
> LogFactory. With this type of the complex entanglement it would seem
> unlikely that these objects would ever be collected. The other
> pattern also includes the spiders web of references. It also
> includes UberspecImpl and a whole bunch of static collections. IME,
> static collections are involved in the vast majority of leaks I've
> Interestingly enough the a portion of the 2nd largest consumer of
> memory is also tangled up in the JDK12Hooks. Random sampling leads
> me to AST parse trees and "no reference". Looks like much of this is
> tied up with Velocity. In fact the largest consumer of memory at 24%
> is char. I'm failing to find anything that is not tied up with
> Velocity (AST parsing).
> Needs more investigation. Be interesting to run a test with
> generations turned on. NetBeans generations is a true count unlike
> that provided by YourKit.
> Jed Wesley-Smith wrote:
>> Classes as well. We end up getting an OOME although the profilers
>> report only a third of the heap is reachable.
>> Although I indicated we saw this on the IBM jdk analysis of that
>> dump showed a completely different issue that apparently may not be
>> a problem (due to reflection optimisation on that jdk) - the dead
>> objects appear to have been correctly cleared. We are reproducing
>> this to verify.
>> Additionally we tried running with -client on the sun jvms as we
>> saw a bug that might have caused it reported against server only
>> but without success.
>> On 16/04/2009, at 12:51 AM, Tony Printezis <Antonios.Printezis at sun.com
>> > wrote:
>>> OK, I'll bite.
>>> When you say: "a large section of memory (a plugin framework)" do
>>> you mean only objects in the young / old gen, or also classes in
>>> the perm gen?
>>> How do you know that said memory is not being reclaimed? Do you
>>> eventually get an OOM?
>>> Given that it happens with two different JVMs (I assume you use
>>> HotSpot on Linux and Mac, as well as the IBM JDK), it's unlikely
>>> to be a GC bug, as both JVMs would need to have the same bug. Not
>>> impossible, but unlikely, IMHO.
>>> Jed Wesley-Smith wrote:
>>>> I am writing to this list in some desperation hoping for some
>>>> expert advice. We (the JIRA development team at Atlassian) have
>>>> been hunting memory leaks for some weeks and in the process have
>>>> tracked down and removed every possible reference to a large
>>>> section of memory (a plugin framework) that we could find.
>>>> Starting with all strong references and proceeding to remove soft
>>>> and weak references - even things like clearing the
>>>> java.lang.reflect.Proxy cache - and even Finalizer references
>>>> until both YourKit, Eclipse MAT, JProfiler and jhat all report
>>>> that the memory in question is dead and should be collectable,
>>>> but inexplicably _the JVM still holds on to it_. There are no JNI
>>>> Global references either, yet this memory remains uncollectable!
>>>> This happens for the 1.5 and 1.6 JVMs on Linux and Mac, and the
>>>> IBM 1.6 JDK on Linux.
>>>> So my question is, how on earth do I search for what is
>>>> referencing this uncollectable memory? Are there any other tools
>>>> that can help find why this memory is not collected? Can I query
>>>> the VM directly somehow?
>>>> I fear this is a JVM GC bug as no known memory analysis tool can
>>>> find the heap root (i.e. according to "the rules" there is no
>>>> heap root). Are there any known GC memory leaks caused by
>>>> ClassLoaders being dropped for instance?
>>>> The application is creating and disposing of a lot of
>>>> ClassLoaders via OSGi (Apache Felix) with Spring OSGi. It creates
>>>> a lot of java.lang.reflect.Proxy class instances.
>>>> We have written this up and added an example heap dump here:
>>>> Having come to the end of our tethers here, if anyone can help in
>>>> any way it would be massively appreciated.
>>>> Jed Wesley-Smith
>>>> JIRA Team @ Atlassian
>>> | Tony Printezis, Staff Engineer | Sun Microsystems
>>> Inc. |
>>> | | MS
>>> UBUR02-311 |
>>> | e-mail: tony.printezis at sun.com | 35 Network
>>> Drive |
>>> | office: +1 781 442 0998 (x20998) | Burlington, MA 01803-2756,
>>> USA |
>>> e-mail client: Thunderbird (Linux)
More information about the hotspot-gc-dev