RFR(s): 8077276: allocating heap with UseLargePages and HugeTLBFS may trash existing memory mappings (linux)

[Thomas Stüfe]

Hi all,

please review this fix to huge page allocation on Linux.

bug: https://bugs.openjdk.java.net/browse/JDK-8077276
webrev: http://cr.openjdk.java.net/~stuefe/webrevs/8077276/webrev.00/webrev/

os::Linux::reserve_memory_special_huge_tlbfs_mixed() first establishes a
mapping with small pages over the whole requested range, then exchanges the
parts of the mapping which are aligned to large page size with large pages.

Now, when establishing the first mapping, it uses os::reserve_memory() with
a potentially non-null req_addr. In that case, os::reserve_memory() will do
a mmap(MAP_FIXED).

This will trash any pre-existing mappings.

Note that I could not willingly reproduce the error with an unmodified VM.
But I added a reproduction case which demonstrated that if one were to call
os::Linux::reserve_memory_special_huge_tlbfs_mixed() with a non-NULL
req_addr, existing mappings would be trashed. Depending on where we are in
address space, we also would overwrite libc structures and crash
immediately.

The repro case is part of the change, see changes in
test_reserve_memory_special_huge_tlbfs_mixed(), and can be executed with:

./images/jdk/bin/java -XX:+UseLargePages -XX:+UseHugeTLBFS -XX:-UseSHM
-XX:+ExecuteInternalVMTests -XX:+VerboseInternalVMTests

The fix: instead of using os::reserve_memory(),
os::Linux::reserve_memory_special_huge_tlbfs_mixed() now calls mmap()
directly with the non-NULL req_addr, but without MAP_FIXED. This means the
OS may do its best to allocate at req_addr, but will not trash any
mappings. This also follows the pattern in
os::Linux::reserve_memory_special_huge_tlbfs_only(), which is a sister
function of os::Linux::reserve_memory_special_huge_tlbfs_mixed().

Note also discussion at:
http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-April/017823.html.

Thanks for reviewing!

Kind Regards, Thomas