Question about stack overflows in native code

Thomas Stüfe thomas.stuefe at
Tue Apr 4 08:26:27 UTC 2017

Hi Fred,

On Mon, Apr 3, 2017 at 8:56 PM, Frederic Parain <frederic.parain at>

> The yellow zone is disabled in the logic executed by the signal handler,
> so yes, if the signal handler doesn't have enough stack space to run,
> the consequence is the immediate death of the process.
thanks for clarifying this!

> In the past, Hotspot was using an alternate stack for signal handlers,
> but it was causing so many issues that the alternate stack support
> has been removed.

Ah.. ok, this would have been my proposal :)

We played with this on AIX in the far past (~10 years ago) but never got it
to be stable. I was hoping the situation would be better now, but I also
realize we do a number of undocumented things in signal handler, especially
jumping to stubs via pc-in-context-modification.

> So the solution today to make a native code friendly with the stack
> overflow handling mechanism, is to incorporate stack banging in it,
> as you have figured out.

See, I wondered whether we could extend the JNI spec for it and provide a
tiny JNI helper function like this:


which would check for required stack size and rise a StackOverflow
Exception if remaining stack is too small. Caller then would have to check
for pending exception and if one is pending, return immediately.

This would of course require cooperation by the native programmers. But
programmers could add calls to this function in code which is clearly

> The size of different zones defined in HotSpot can help you sizing
> the stack banging size.
> The size of the red zone is the stack size required to generated the
> crash report (hs_error file).
> The size of the yellow zone is the stack size required to throw a
> StackOverflowError.
> The size of shadow pages is the stack size required to safely perform
> a VM call or a call to native method of the JDK.
> Hope this helps.
Sure! Thanks, Thomas

> Fred
> On 04/03/2017 02:42 PM, Thomas Stüfe wrote:
>> Hi Fred,
>> thanks! Some more questions inline.
>> On Mon, Apr 3, 2017 at 8:29 PM, Frederic Parain
>> <frederic.parain at <mailto:frederic.parain at>> wrote:
>>     When the yellow zone is hit and the thread state is not in
>>     _thread_in_java (which means thread state is _thread_in_native or
>>     _thread_in_vm), the yellow zone is silently disabled and the thread
>>     is allowed to resume its execution.
>> Disabled by whom exactly?
>> Normally, this would be done in the signal handler, but that requires
>> enough stack space to run. AFAIK jitted or interpreted code does stack
>> banging in order to trigger the yellow-page-segfault at a point where
>> there are enough pages left on the stack to invoke the signal handler (n
>> shadow pages before), but that is not guaranteed to work with native
>> C-compiled code, no?
>> (not just a theory, we have a test case here where a stack overflow in
>> native code just silently kills the process.)
>> I guess it may work accidentally if the C-compiled code itself does some
>> form of stack banging when establishing frames, in order to detect OS
>> stack overflows? Very fuzzy here. But whatever the C-compiled code does,
>> it has no notion about how much space we need to invoke the signal
>> handler and handle stack overflows, no?
>>     When the red zone is hit, what ever the current thread state is,
>>     the red zone is disabled and VMError::report_and_die() is called,
>>     which should generate a hs_err file unless the generation of the
>>     error file requires more memory than the red zone provides.
>>     Fred
>> Thanks, Thomas
>>     On 04/03/2017 02:08 PM, Thomas Stüfe wrote:
>>         Hi,
>>         Today we wondered what would happen when a stack overflow occurs
>>         in native
>>         code running in a java thread (an attached thread or one created
>>         by the VM).
>>         In that case yellow and red pages are in place, but this would
>>         not help
>>         much, would it not, because the native code would not do any
>>         stack banging?
>>         So, native code would hit the yellow page, and then there would
>>         probably
>>         not be enough space left on the stack to invoke the signal
>>         handler. The
>>         result would be immediate VM death - not even an hs-err file -
>>         is that
>>         correct?
>>         Also, we would hit the our own yellow page, not the guard page
>>         the OS may
>>         or may not have established, so - on UNIX - this would show up as
>>         "Segmentation Fault", not "Stack Overflow", or?
>>         Thank you,
>>         Thomas

More information about the hotspot-runtime-dev mailing list