JVM hangs beyond recovery
David.Holmes at oracle.com
Wed Jun 9 02:13:05 PDT 2010
I can't shed any further light on this as there are too many
unidentified frames in the stack traces. From a quick read the MySQL
connector is supposed to be pure Java, so that would rule it out.
One thing odd I did notice in the stack dump is thread 83:
#7 0x00002b4b2f9d4e2e in JVM_MonitorWait ()
#8 0x00002aaaaaf73986 in ?? ()
#9 0x0000000000000003 in ?? ()
#10 0x00002b4b2fb6a023 in os::javaTimeMillis ()
#11 0x00002aaaaaf67a42 in ?? ()
#12 0x0000000000000000 in ?? ()
There's no way to get from os::javaTimeMillis into JVM_MonitorWait.
Stas Oskin said the following on 06/09/10 18:36:
> Thanks for the explanation.
> The underlying problem looks familiar to me though I can't find the
> culprit thread in the gdb thread dump. This problem occurs when a
> library uses a native dynamic loader hook (I'll call it dlsym) to
> make a call into the JVM. This is done while holding an internal
> dlsym mutex lock and the end result is a deadlock if any of the
> other threads in the Java application need to perform a dlsym
> related operation (which you can see in the thread dump that they do
> and are all blocked on that internal mutex).
> I noticed this happening mostly in DB related operations. I'm using
> mysql-connector-java-5.1.6 which according to MySQL documentation, is "a
> native Java driver", and indeed doesn't seem to contain any native code
> library, but I wonder if this related to what you experienced in past.
> The culprit is whomever calls into the VM from inside the dlsym
> code. As I said I can't see that culprit in the stack traces, but
> typically it happens when someone installs an "on-load" hook or
> something of that nature.
> Is there any good way to detect the exact library causing this deadlock?
> This happens quite rarely and spontaneously, so just disabling native
> libraries one by one and trying to find the cause didn't work for me.
> Thanks again for all the help.
More information about the hotspot-runtime-dev