the amazing tales of the search for the invisible man! or, where's my gc root

Jed Wesley-Smith jed at
Fri Apr 17 23:32:25 UTC 2009


Thanks for the info. I will try NetBeans and look at what it tells me  
- this is certainly a lot more useful than what the other tools have  
been able to give me. Unfortunately it is Saturday morning in Sydney  
and I will not be able to attempt with generations on until Monday.

As reported below, we have verified that the IBM JVM does not leak the  
plugin system classes.


On 18/04/2009, at 3:51 AM, kirk wrote:

> Hi Jed,
> I've had a quick look at the heap dump. I'm having a little trouble  
> understanding what is in there. What I can see is a large number of  
> java.lang.reflect.Method objects being held. There seems to be two  
> competing patterns of references holding onto these objects. I've  
> attached some screenshots rather than use words.
> The scary thing is that the references include ClassLoader.scl,  
> JDK12Hooks.systemClassLoader as well as Apache commons logging  
> LogFactory. With this type of the complex entanglement it would seem  
> unlikely that these objects would ever be collected. The other  
> pattern also includes the spiders web of references. It also  
> includes UberspecImpl and a whole bunch of static collections. IME,  
> static collections are involved in the vast majority of leaks I've  
> diagnosed.
> Interestingly enough the a portion of the 2nd largest consumer of  
> memory is also tangled up in the JDK12Hooks. Random sampling leads  
> me to AST parse trees and "no reference". Looks like much of this is  
> tied up with Velocity. In fact the largest consumer of memory at 24%  
> is char[]. I'm failing to find anything that is not tied up with  
> Velocity (AST parsing).
> Needs more investigation. Be interesting to run a test with  
> generations turned on. NetBeans generations is a true count unlike  
> that provided by YourKit.
> Regards,
> Kirk
> Jed Wesley-Smith wrote:
>> Classes as well. We end up getting an OOME although the profilers  
>> report only a third of the heap is reachable.
>> Although I indicated we saw this on the IBM jdk analysis of that  
>> dump showed a completely different issue that apparently may not be  
>> a problem (due to reflection optimisation on that jdk) - the dead  
>> objects appear to have been correctly cleared. We are reproducing  
>> this to verify.
>> Additionally we tried running with -client on the sun jvms as we  
>> saw a bug that might have caused it reported against server only  
>> but without success.
>> cheers,
>> jed.
>> On 16/04/2009, at 12:51 AM, Tony Printezis <Antonios.Printezis at 
>> > wrote:
>>> OK, I'll bite.
>>> When you say: "a large section of memory (a plugin framework)" do  
>>> you mean only objects in the young / old gen, or also classes in  
>>> the perm gen?
>>> How do you know that said memory is not being reclaimed? Do you  
>>> eventually get an OOM?
>>> Given that it happens with two different JVMs (I assume you use  
>>> HotSpot on Linux and Mac, as well as the IBM JDK), it's unlikely  
>>> to be a GC bug, as both JVMs would need to have the same bug. Not  
>>> impossible, but unlikely, IMHO.
>>> Tony
>>> Jed Wesley-Smith wrote:
>>>> all,
>>>> I am writing to this list in some desperation hoping for some  
>>>> expert advice. We (the JIRA development team at Atlassian) have  
>>>> been hunting memory leaks for some weeks and in the process have  
>>>> tracked down and removed every possible reference to a large  
>>>> section of memory (a plugin framework) that we could find.  
>>>> Starting with all strong references and proceeding to remove soft  
>>>> and weak references - even things like clearing the  
>>>> java.lang.reflect.Proxy cache - and even Finalizer references  
>>>> until both YourKit, Eclipse MAT, JProfiler and jhat all report  
>>>> that the memory in question is dead and should be collectable,  
>>>> but inexplicably _the JVM still holds on to it_. There are no JNI  
>>>> Global references either, yet this memory remains uncollectable!
>>>> This happens for the 1.5 and 1.6 JVMs on Linux and Mac, and the  
>>>> IBM 1.6 JDK on Linux.
>>>> So my question is, how on earth do I search for what is  
>>>> referencing this uncollectable memory? Are there any other tools  
>>>> that can help find why this memory is not collected? Can I query  
>>>> the VM directly somehow?
>>>> I fear this is a JVM GC bug as no known memory analysis tool can  
>>>> find the heap root (i.e. according to "the rules" there is no  
>>>> heap root). Are there any known GC memory leaks caused by  
>>>> ClassLoaders being dropped for instance?
>>>> The application is creating and disposing of a lot of  
>>>> ClassLoaders via OSGi (Apache Felix) with Spring OSGi. It creates  
>>>> a lot of java.lang.reflect.Proxy class instances.
>>>> We have written this up and added an example heap dump here:
>>>> Having come to the end of our tethers here, if anyone can help in  
>>>> any way it would be massively appreciated.
>>>> cheers,
>>>> Jed Wesley-Smith
>>>> JIRA Team @ Atlassian
>>> -- 
>>> ---------------------------------------------------------------------
>>> | Tony Printezis, Staff Engineer   | Sun Microsystems  
>>> Inc.          |
>>> |                                  | MS  
>>> UBUR02-311                  |
>>> | e-mail: tony.printezis at   | 35 Network  
>>> Drive               |
>>> | office: +1 781 442 0998 (x20998) | Burlington, MA 01803-2756,  
>>> USA |
>>> ---------------------------------------------------------------------
>>> e-mail client: Thunderbird (Linux)

More information about the hotspot-gc-dev mailing list