RFR(L) 8046598: Scalable Native memory tracking development

Zhengyu Gu zhengyu.gu at oracle.com
Tue Jul 15 20:12:04 UTC 2014

This is an update to previous RFR 8028541: Native Memory Tracking 
enhancement, the original one is closed as duplicate of current one.

The update is mainly based on feedback from Coleen and Christian:
   - Refactored MemReporter to break up some large functions and 
eliminate duplicated code.
   - Minor change to MemBaseline for eliminating duplicated code.
   - Changed MEMFLAGS type from unsigned  short => MemoryType.

Also added unit tests for LinkedList.

The note from  RFR 8028541:
This is significant rework of native memory tracking introduced in 
earlier releases.

The goal of this enhancement is to improve scalability, from both 
tracking memory and CPU usage perspectives, so it can scale well with 
increased memory allocation in large applications.

The enhancement is mainly focused on malloc memory tracking, whose 
activities are several magnitude higher than virtual memory, and was the 
main bottleneck in early implementation.

Instead of using book keeping records for tracking malloc activities, 
new implementation co-locates tracking data along side with user data by 
using a prefixed header. The header size is 8 bytes on 32-bit systems 
and 16 bytes on 64-bit systems, which ensure that user data also align 

Virtual memory tracking still uses book keeping records, and 
ThreadCritical lock is always acquired to alter the records and related 
data structures.

Summary tracking data is maintained in static data structures, via 
atomic operations. Malloc detail tracking call stacks are maintained in 
a lock free hashtable.

The key improvements:
   1. Up-to-date tracking report.
   2. Detail tracking now shows multiple call frames. Number of frames 
is compilation time decision, currently default to 4.
   3. Malloc tracking is lock free.
   4. Tracking summary is reported in hs_err file when native memory 
tracking is enabled.
   5. Query is faster, uses little memory and need a very little process.

The drawback is that, malloc tracking header is always needed if native 
memory tracking has ever been enabled, even after tracking is shutdown.

   The most noticeable impact for JVM developers, is that Arena now also 
take memory type as constructor parameter, besides the new operators.
      Arena* a = new (mtCode) Arena()  => Arena* a = new (mtCode) 

The webrev shows modification of about 60 files, but most of them are 
due to tracking API changes, mainly due to tracking stack, now, is an 
object, vs. a single pc.

The most important files for this implementations are:

mallocTracker.hpp/cpp and mallocTracker.inline.hpp


   - JPRT
   - NMT test suite
   - vm.quick.testlist
   - Kitchensink stability test for 16+ days
   - FMW

Bug: https://bugs.openjdk.java.net/browse/JDK-8046598
Webrev: http://cr.openjdk.java.net/~zgu/8046598/webrev.00/ 



More information about the hotspot-dev mailing list