From vladimir.kozlov at oracle.com  Mon Aug  2 15:01:02 2010
From: vladimir.kozlov at oracle.com (Vladimir Kozlov)
Date: Mon, 02 Aug 2010 15:01:02 -0700
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in ciBlock::start_bci()
	with EA]
Message-ID: <4C57401E.5040204@oracle.com>

Forwarding to GC and Runtime groups since it is common code.

Vladimir

-------- Original Message --------
Subject: Request for reviews (M): 6973963: SEGV in ciBlock::start_bci() with EA
Date: Mon, 02 Aug 2010 14:57:25 -0700
From: Vladimir Kozlov <vladimir.kozlov at oracle.com>
To: hotspot compiler <hotspot-compiler-dev at openjdk.java.net>

http://cr.openjdk.java.net/~kvn/6973963/webrev

Fixed 6973963: SEGV in ciBlock::start_bci() with EA

I added stress recompilation during CompileTheWorld and found this case.
It is similar to 6968368. BCEscapeAnalyzer::do_analysis() calls
ciMethod::get_method_blocks() which calls constructor ciMethodBlocks.
This constructor allocates GrowableArray elements on stack (thread
local resource area). As result when the method recompiled without EA
_blocks->_data is NULL.

Solution:
Added stress recompilation during CompileTheWorld: recompile with
subsume_loads = false and do_escape_analysis = false.
Added more checks into ResourceObj and growableArray to verify correctness
of allocation. I have to relax the new assert in GrowableArray when
elements are allocated on arena to allow allocattion of GrowableArray object
as a part of an other object (for example, in ConnectionGraph and SuperWord).

Tested with failed cases, CTW.

From vladimir.kozlov at oracle.com  Mon Aug  2 15:39:00 2010
From: vladimir.kozlov at oracle.com (Vladimir Kozlov)
Date: Mon, 02 Aug 2010 15:39:00 -0700
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>
References: <4C57401E.5040204@oracle.com>
	<C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>
Message-ID: <4C574904.2050307@oracle.com>

Thank you, Tom

RESOURCE_AREA case combines 3 cases:

GrowableArray<int> *foo = new GrowableArray<int>(size); // resource array allocation

GrowableArray<int> foo; // stack allocation

class A : public ResourceObj {
   GrowableArray<int> foo; // embedding allocation
}

In all this cases GrowableArray::_data array is allocated in thread local
resource area. GrowableArray calls all these cases as stack allocation
and I followed this naming.

But you are right I can separate 2 last cases into "stack" (stack or embedding)
allocation, I still have 1 bit in allocation_type :)

Thanks,
Vladimir

Tom Rodriguez wrote:
> So doesn't this triple the amount of time taken by CTW?  That seems kind of extreme for the default mode.  Maybe it should be under a flag?  The allocation.hpp changes look a little sketchy.  
> 
> +     if (~(_allocation | allocation_mask) != (uintptr_t)this) {
> +       set_allocation_type((address)this, RESOURCE_AREA);
> 
> Why is this case called RESOURCE_AREA?  Isn't this the stack or embedding case?
> 
> This doesn't make sense either.
> 
> !   bool allocated_on_stack()     { return get_allocation_type() == RESOURCE_AREA; }
> 
> Anyway, the existing logic around this seemed sketchy so I can't quite say whether this is better or not.  I'll have to leave that to someone else.
> 
> The GrowableArray changes themselves look fine.
> 
> tom
> 
> On Aug 2, 2010, at 3:01 PM, Vladimir Kozlov wrote:
> 
>> Forwarding to GC and Runtime groups since it is common code.
>>
>> Vladimir
>>
>> -------- Original Message --------
>> Subject: Request for reviews (M): 6973963: SEGV in ciBlock::start_bci() with EA
>> Date: Mon, 02 Aug 2010 14:57:25 -0700
>> From: Vladimir Kozlov <vladimir.kozlov at oracle.com>
>> To: hotspot compiler <hotspot-compiler-dev at openjdk.java.net>
>>
>> http://cr.openjdk.java.net/~kvn/6973963/webrev
>>
>> Fixed 6973963: SEGV in ciBlock::start_bci() with EA
>>
>> I added stress recompilation during CompileTheWorld and found this case.
>> It is similar to 6968368. BCEscapeAnalyzer::do_analysis() calls
>> ciMethod::get_method_blocks() which calls constructor ciMethodBlocks.
>> This constructor allocates GrowableArray elements on stack (thread
>> local resource area). As result when the method recompiled without EA
>> _blocks->_data is NULL.
>>
>> Solution:
>> Added stress recompilation during CompileTheWorld: recompile with
>> subsume_loads = false and do_escape_analysis = false.
>> Added more checks into ResourceObj and growableArray to verify correctness
>> of allocation. I have to relax the new assert in GrowableArray when
>> elements are allocated on arena to allow allocattion of GrowableArray object
>> as a part of an other object (for example, in ConnectionGraph and SuperWord).
>>
>> Tested with failed cases, CTW.
> 

From john.cuthbertson at sun.com  Mon Aug  2 16:52:41 2010
From: john.cuthbertson at sun.com (john.cuthbertson at sun.com)
Date: Mon, 02 Aug 2010 23:52:41 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6814437: G1: remove the _new_refs array
Message-ID: <20100802235243.867C747E7B@hg.openjdk.java.net>

Changeset: 2d160770d2e5
Author:    johnc
Date:      2010-08-02 12:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/2d160770d2e5

6814437: G1: remove the _new_refs array
Summary: The per-worker _new_refs array is used to hold references that point into the collection set. It is populated during RSet updating and subsequently processed. In the event of an evacuation failure it processed again to recreate the RSets of regions in the collection set. Remove the per-worker _new_refs array by processing the references directly. Use a DirtyCardQueue to hold the cards containing the references so that the RSets of regions in the collection set can be recreated when handling an evacuation failure.
Reviewed-by: iveresov, jmasa, tonyp

! src/share/vm/gc_implementation/g1/concurrentG1Refine.cpp
! src/share/vm/gc_implementation/g1/concurrentG1Refine.hpp
! src/share/vm/gc_implementation/g1/dirtyCardQueue.cpp
! src/share/vm/gc_implementation/g1/dirtyCardQueue.hpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp
! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp
! src/share/vm/gc_implementation/g1/g1CollectorPolicy.hpp
! src/share/vm/gc_implementation/g1/g1OopClosures.inline.hpp
! src/share/vm/gc_implementation/g1/g1RemSet.cpp
! src/share/vm/gc_implementation/g1/g1RemSet.hpp
! src/share/vm/gc_implementation/g1/g1RemSet.inline.hpp
! src/share/vm/gc_implementation/g1/heapRegion.cpp
! src/share/vm/gc_implementation/includeDB_gc_g1


From vladimir.kozlov at oracle.com  Mon Aug  2 22:45:10 2010
From: vladimir.kozlov at oracle.com (Vladimir Kozlov)
Date: Mon, 02 Aug 2010 22:45:10 -0700
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <4C574904.2050307@oracle.com>
References: <4C57401E.5040204@oracle.com>
	<C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>
	<4C574904.2050307@oracle.com>
Message-ID: <4C57ACE6.6090702@oracle.com>

I updated webrev:

http://cr.openjdk.java.net/~kvn/6973963/webrev.01

Additional changes:

1. Used new option StressRecompilation instead of CompileTheWorld.
    I am still not sure if it is OK since we never test our stress options.

2. Added new allocation type: STACK_OR_EMBEDDED for cases when new() is not called.

3. Added ResourceObj destructor to zap _allocation field.

4. Added assert into get_allocation_type() to check that 'this' address is still encoded in _allocation.
    Found several cases where it was not true, have to add copy constructor and assignment operator.

5. Moved all new methods with asserts into allocation.cpp.

6. The added assert failed for CodeBuffer since it destroys itself inside destructor before ResourceObj destructor called.
    Moved Copy::fill_to_bytes(badResourceValue) into CodeBuffer::operator delete().

7. Replaced PhaseCFG::_node_latency field with pointer since it is valid only inside resource mark in GlobalCodeMotion().

Thanks,
Vladimir

Vladimir Kozlov wrote:
> Thank you, Tom
> 
> RESOURCE_AREA case combines 3 cases:
> 
> GrowableArray<int> *foo = new GrowableArray<int>(size); // resource 
> array allocation
> 
> GrowableArray<int> foo; // stack allocation
> 
> class A : public ResourceObj {
>   GrowableArray<int> foo; // embedding allocation
> }
> 
> In all this cases GrowableArray::_data array is allocated in thread local
> resource area. GrowableArray calls all these cases as stack allocation
> and I followed this naming.
> 
> But you are right I can separate 2 last cases into "stack" (stack or 
> embedding)
> allocation, I still have 1 bit in allocation_type :)
> 
> Thanks,
> Vladimir
> 
> Tom Rodriguez wrote:
>> So doesn't this triple the amount of time taken by CTW?  That seems 
>> kind of extreme for the default mode.  Maybe it should be under a 
>> flag?  The allocation.hpp changes look a little sketchy. 
>> +     if (~(_allocation | allocation_mask) != (uintptr_t)this) {
>> +       set_allocation_type((address)this, RESOURCE_AREA);
>>
>> Why is this case called RESOURCE_AREA?  Isn't this the stack or 
>> embedding case?
>>
>> This doesn't make sense either.
>>
>> !   bool allocated_on_stack()     { return get_allocation_type() == 
>> RESOURCE_AREA; }
>>
>> Anyway, the existing logic around this seemed sketchy so I can't quite 
>> say whether this is better or not.  I'll have to leave that to someone 
>> else.
>>
>> The GrowableArray changes themselves look fine.
>>
>> tom
>>
>> On Aug 2, 2010, at 3:01 PM, Vladimir Kozlov wrote:
>>
>>> Forwarding to GC and Runtime groups since it is common code.
>>>
>>> Vladimir
>>>
>>> -------- Original Message --------
>>> Subject: Request for reviews (M): 6973963: SEGV in 
>>> ciBlock::start_bci() with EA
>>> Date: Mon, 02 Aug 2010 14:57:25 -0700
>>> From: Vladimir Kozlov <vladimir.kozlov at oracle.com>
>>> To: hotspot compiler <hotspot-compiler-dev at openjdk.java.net>
>>>
>>> http://cr.openjdk.java.net/~kvn/6973963/webrev
>>>
>>> Fixed 6973963: SEGV in ciBlock::start_bci() with EA
>>>
>>> I added stress recompilation during CompileTheWorld and found this case.
>>> It is similar to 6968368. BCEscapeAnalyzer::do_analysis() calls
>>> ciMethod::get_method_blocks() which calls constructor ciMethodBlocks.
>>> This constructor allocates GrowableArray elements on stack (thread
>>> local resource area). As result when the method recompiled without EA
>>> _blocks->_data is NULL.
>>>
>>> Solution:
>>> Added stress recompilation during CompileTheWorld: recompile with
>>> subsume_loads = false and do_escape_analysis = false.
>>> Added more checks into ResourceObj and growableArray to verify 
>>> correctness
>>> of allocation. I have to relax the new assert in GrowableArray when
>>> elements are allocated on arena to allow allocattion of GrowableArray 
>>> object
>>> as a part of an other object (for example, in ConnectionGraph and 
>>> SuperWord).
>>>
>>> Tested with failed cases, CTW.
>>

From vladimir.kozlov at oracle.com  Tue Aug  3 02:33:28 2010
From: vladimir.kozlov at oracle.com (Vladimir Kozlov)
Date: Tue, 03 Aug 2010 02:33:28 -0700
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <4C57B5D0.4090807@oracle.com>
References: <4C57401E.5040204@oracle.com>	<C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>	<4C574904.2050307@oracle.com>
	<4C57ACE6.6090702@oracle.com> <4C57B5D0.4090807@oracle.com>
Message-ID: <4C57E268.5060602@oracle.com>

Thank you, Dave

On 8/2/10 11:23 PM, David Holmes wrote:
> Hi Vladimir,
>
> I always get nervous when people start tweaking these fundamental allocation classes - there are always unforeseen
> interactions. :)

I hate to touch it also. But my patience was broken when I found this problem.

>
> This is only a partial review.
>
> src/share/vm/asm/codeBuffer.cpp
>
> + void CodeBuffer::operator delete(void* p) {
> + ResourceObj::operator delete(p);
> #ifdef ASSERT
> ! Copy::fill_to_bytes(p, sizeof(CodeBuffer), badResourceValue);
> #endif
> }
>
> You moved the copy from the destructor to operator delete, but now you access p after you have deleted it.

Yes, you are right. But CodeBuffer is never allocated on C heap and ResourceObj::delete(p)
should be called only for allocation on C_HEAP. So this code should never be executed,
I will replace it with ShouldNotCallThis() in CodeBuffer::delete(void* p).
Which leaves the original problem: I need to find how to zap CodeBuffer after ResourceObj destructor
which is called after CodeBuffer destructor.
Note: the problem here is not CodeBuffer but its field: OopRecorder  _default_oop_recorder;

>
> src/share/vm/utilities/growableArray.hpp
>
> + // on stack or ebedded into an other object.
>
> Typo: embedded

OK.

>
> src/share/vm/memory/allocation.hpp
>
> ! ~ResourceObj();
>
> You've added a destructor, but subclasses define their own destructors - so doesn't this need to be virtual?

Constructor and destructor are special methods. Super class's default destructor are always called from
subclass's destructor. The only case when you need to declare it as virtual if subclass object is assigned to
local/field of super class type and compiler does not know what the original subclass was.

ResourceObj *f = new OopRecorder();
...
delete f;

We don't use such constructions.

>
> src/share/vm/memory/allocation.cpp
>
> + #ifdef ASSERT
> + ((ResourceObj *)p)->_allocation = badHeapOopVal;
> + #endif // ASSERT
>
> For consistency this should be a DEBUG_ONLY(...)

OK. I had problems recently with some macros when I passed expressions with nested () so I was too conservative here.

>
> + assert((allocation & allocation_mask) == 0, "");
> + assert(type <= allocation_mask, "");
>
> Please give meaningful messages to assertion failures.

OK. I will update webrev today (during work hours :) ).

Thanks,
Vladimir

>
> David
> -----
>
> Vladimir Kozlov said the following on 08/03/10 15:45:
>> I updated webrev:
>>
>> http://cr.openjdk.java.net/~kvn/6973963/webrev.01
>>
>> Additional changes:
>>
>> 1. Used new option StressRecompilation instead of CompileTheWorld.
>> I am still not sure if it is OK since we never test our stress options.
>>
>> 2. Added new allocation type: STACK_OR_EMBEDDED for cases when new() is not called.
>>
>> 3. Added ResourceObj destructor to zap _allocation field.
>>
>> 4. Added assert into get_allocation_type() to check that 'this' address is still encoded in _allocation.
>> Found several cases where it was not true, have to add copy constructor and assignment operator.
>>
>> 5. Moved all new methods with asserts into allocation.cpp.
>>
>> 6. The added assert failed for CodeBuffer since it destroys itself inside destructor before ResourceObj destructor
>> called.
>> Moved Copy::fill_to_bytes(badResourceValue) into CodeBuffer::operator delete().
>>
>> 7. Replaced PhaseCFG::_node_latency field with pointer since it is valid only inside resource mark in GlobalCodeMotion().
>>
>> Thanks,
>> Vladimir
>>
>> Vladimir Kozlov wrote:
>>> Thank you, Tom
>>>
>>> RESOURCE_AREA case combines 3 cases:
>>>
>>> GrowableArray<int> *foo = new GrowableArray<int>(size); // resource array allocation
>>>
>>> GrowableArray<int> foo; // stack allocation
>>>
>>> class A : public ResourceObj {
>>> GrowableArray<int> foo; // embedding allocation
>>> }
>>>
>>> In all this cases GrowableArray::_data array is allocated in thread local
>>> resource area. GrowableArray calls all these cases as stack allocation
>>> and I followed this naming.
>>>
>>> But you are right I can separate 2 last cases into "stack" (stack or embedding)
>>> allocation, I still have 1 bit in allocation_type :)
>>>
>>> Thanks,
>>> Vladimir
>>>
>>> Tom Rodriguez wrote:
>>>> So doesn't this triple the amount of time taken by CTW? That seems kind of extreme for the default mode. Maybe it
>>>> should be under a flag? The allocation.hpp changes look a little sketchy. + if (~(_allocation | allocation_mask) !=
>>>> (uintptr_t)this) {
>>>> + set_allocation_type((address)this, RESOURCE_AREA);
>>>>
>>>> Why is this case called RESOURCE_AREA? Isn't this the stack or embedding case?
>>>>
>>>> This doesn't make sense either.
>>>>
>>>> ! bool allocated_on_stack() { return get_allocation_type() == RESOURCE_AREA; }
>>>>
>>>> Anyway, the existing logic around this seemed sketchy so I can't quite say whether this is better or not. I'll have
>>>> to leave that to someone else.
>>>>
>>>> The GrowableArray changes themselves look fine.
>>>>
>>>> tom
>>>>
>>>> On Aug 2, 2010, at 3:01 PM, Vladimir Kozlov wrote:
>>>>
>>>>> Forwarding to GC and Runtime groups since it is common code.
>>>>>
>>>>> Vladimir
>>>>>
>>>>> -------- Original Message --------
>>>>> Subject: Request for reviews (M): 6973963: SEGV in ciBlock::start_bci() with EA
>>>>> Date: Mon, 02 Aug 2010 14:57:25 -0700
>>>>> From: Vladimir Kozlov <vladimir.kozlov at oracle.com>
>>>>> To: hotspot compiler <hotspot-compiler-dev at openjdk.java.net>
>>>>>
>>>>> http://cr.openjdk.java.net/~kvn/6973963/webrev
>>>>>
>>>>> Fixed 6973963: SEGV in ciBlock::start_bci() with EA
>>>>>
>>>>> I added stress recompilation during CompileTheWorld and found this case.
>>>>> It is similar to 6968368. BCEscapeAnalyzer::do_analysis() calls
>>>>> ciMethod::get_method_blocks() which calls constructor ciMethodBlocks.
>>>>> This constructor allocates GrowableArray elements on stack (thread
>>>>> local resource area). As result when the method recompiled without EA
>>>>> _blocks->_data is NULL.
>>>>>
>>>>> Solution:
>>>>> Added stress recompilation during CompileTheWorld: recompile with
>>>>> subsume_loads = false and do_escape_analysis = false.
>>>>> Added more checks into ResourceObj and growableArray to verify correctness
>>>>> of allocation. I have to relax the new assert in GrowableArray when
>>>>> elements are allocated on arena to allow allocattion of GrowableArray object
>>>>> as a part of an other object (for example, in ConnectionGraph and SuperWord).
>>>>>
>>>>> Tested with failed cases, CTW.
>>>>

From vladimir.kozlov at oracle.com  Tue Aug  3 11:33:38 2010
From: vladimir.kozlov at oracle.com (Vladimir Kozlov)
Date: Tue, 03 Aug 2010 11:33:38 -0700
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <4C5807F1.1070607@oracle.com>
References: <4C57401E.5040204@oracle.com>	<C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>	<4C574904.2050307@oracle.com>
	<4C57ACE6.6090702@oracle.com> <4C57B5D0.4090807@oracle.com>
	<4C57E268.5060602@oracle.com> <4C5807F1.1070607@oracle.com>
Message-ID: <4C586102.3050508@oracle.com>

I updated webrev

http://cr.openjdk.java.net/~kvn/6973963/webrev.02

I added 'virtual' to ~ResourceObj() to avoid any surprises as Dave pointed.

I save/restore allocation type around Copy::fill_to_bytes() in ~CodeBuffer() to solve my problem.

Vladimir

David Holmes wrote:
> Vladimir Kozlov said the following on 08/03/10 19:33:
>> On 8/2/10 11:23 PM, David Holmes wrote:
>>> src/share/vm/asm/codeBuffer.cpp
>>>
>>> + void CodeBuffer::operator delete(void* p) {
>>> + ResourceObj::operator delete(p);
>>> #ifdef ASSERT
>>> ! Copy::fill_to_bytes(p, sizeof(CodeBuffer), badResourceValue);
>>> #endif
>>> }
>>>
>>> You moved the copy from the destructor to operator delete, but now 
>>> you access p after you have deleted it.
>>
>> Yes, you are right. But CodeBuffer is never allocated on C heap and 
>> ResourceObj::delete(p)
>> should be called only for allocation on C_HEAP. So this code should 
>> never be executed,
> 
> I'm confused. CodeBuffer is a StackObj, not a ResourceObj, so why would 
> you call ResourceObj::delete in the first place ??
> 
>> I will replace it with ShouldNotCallThis() in CodeBuffer::delete(void* 
>> p).
>> Which leaves the original problem: I need to find how to zap 
>> CodeBuffer after ResourceObj destructor
>> which is called after CodeBuffer destructor.
> 
> Again I'm missing something - what is the connection between CodeBuffer 
> and ResourceObj ?
> 
>> Note: the problem here is not CodeBuffer but its field: OopRecorder  
>> _default_oop_recorder;
> 
> Explain that to me off-list if you like, I'm not familiar with this part 
> of the code.
> 
>>> src/share/vm/memory/allocation.hpp
>>>
>>> ! ~ResourceObj();
>>>
>>> You've added a destructor, but subclasses define their own 
>>> destructors - so doesn't this need to be virtual?
>>
>> Constructor and destructor are special methods. Super class's default 
>> destructor are always called from
>> subclass's destructor. The only case when you need to declare it as 
>> virtual if subclass object is assigned to
>> local/field of super class type and compiler does not know what the 
>> original subclass was.
>>
>> ResourceObj *f = new OopRecorder();
>> ...
>> delete f;
>>
>> We don't use such constructions.
> 
> Ok. Just wanted to check. In another codebase we got bitten by a cleanup 
> method in the superclass that did "delete this;" which did not invoke 
> the destructor for the dynamic type of 'this' because the destructor was 
> not virtual.
> 
> Cheers,
> David

From tom.rodriguez at oracle.com  Mon Aug  2 15:24:51 2010
From: tom.rodriguez at oracle.com (Tom Rodriguez)
Date: Mon, 2 Aug 2010 15:24:51 -0700
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <4C57401E.5040204@oracle.com>
References: <4C57401E.5040204@oracle.com>
Message-ID: <C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>

So doesn't this triple the amount of time taken by CTW?  That seems kind of extreme for the default mode.  Maybe it should be under a flag?  The allocation.hpp changes look a little sketchy.  

+     if (~(_allocation | allocation_mask) != (uintptr_t)this) {
+       set_allocation_type((address)this, RESOURCE_AREA);

Why is this case called RESOURCE_AREA?  Isn't this the stack or embedding case?

This doesn't make sense either.

!   bool allocated_on_stack()     { return get_allocation_type() == RESOURCE_AREA; }

Anyway, the existing logic around this seemed sketchy so I can't quite say whether this is better or not.  I'll have to leave that to someone else.

The GrowableArray changes themselves look fine.

tom

On Aug 2, 2010, at 3:01 PM, Vladimir Kozlov wrote:

> Forwarding to GC and Runtime groups since it is common code.
> 
> Vladimir
> 
> -------- Original Message --------
> Subject: Request for reviews (M): 6973963: SEGV in ciBlock::start_bci() with EA
> Date: Mon, 02 Aug 2010 14:57:25 -0700
> From: Vladimir Kozlov <vladimir.kozlov at oracle.com>
> To: hotspot compiler <hotspot-compiler-dev at openjdk.java.net>
> 
> http://cr.openjdk.java.net/~kvn/6973963/webrev
> 
> Fixed 6973963: SEGV in ciBlock::start_bci() with EA
> 
> I added stress recompilation during CompileTheWorld and found this case.
> It is similar to 6968368. BCEscapeAnalyzer::do_analysis() calls
> ciMethod::get_method_blocks() which calls constructor ciMethodBlocks.
> This constructor allocates GrowableArray elements on stack (thread
> local resource area). As result when the method recompiled without EA
> _blocks->_data is NULL.
> 
> Solution:
> Added stress recompilation during CompileTheWorld: recompile with
> subsume_loads = false and do_escape_analysis = false.
> Added more checks into ResourceObj and growableArray to verify correctness
> of allocation. I have to relax the new assert in GrowableArray when
> elements are allocated on arena to allow allocattion of GrowableArray object
> as a part of an other object (for example, in ConnectionGraph and SuperWord).
> 
> Tested with failed cases, CTW.


From David.Holmes at oracle.com  Mon Aug  2 23:23:12 2010
From: David.Holmes at oracle.com (David Holmes)
Date: Tue, 03 Aug 2010 16:23:12 +1000
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <4C57ACE6.6090702@oracle.com>
References: <4C57401E.5040204@oracle.com>	<C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>	<4C574904.2050307@oracle.com>
	<4C57ACE6.6090702@oracle.com>
Message-ID: <4C57B5D0.4090807@oracle.com>

Hi Vladimir,

I always get nervous when people start tweaking these fundamental 
allocation classes - there are always unforeseen interactions. :)

This is only a partial review.

src/share/vm/asm/codeBuffer.cpp

+ void CodeBuffer::operator delete(void* p) {
+   ResourceObj::operator delete(p);
   #ifdef ASSERT
!   Copy::fill_to_bytes(p, sizeof(CodeBuffer), badResourceValue);
   #endif
   }

You moved the copy from the destructor to operator delete, but now you 
access p after you have deleted it.

src/share/vm/utilities/growableArray.hpp

+     // on stack or ebedded into an other object.

Typo: embedded

src/share/vm/memory/allocation.hpp

!   ~ResourceObj();

You've added a destructor, but subclasses define their own destructors - 
so doesn't this need to be virtual?

src/share/vm/memory/allocation.cpp

+ #ifdef ASSERT
+   ((ResourceObj *)p)->_allocation = badHeapOopVal;
+ #endif // ASSERT

For consistency this should be a DEBUG_ONLY(...)

+     assert((allocation & allocation_mask) == 0, "");
+     assert(type <= allocation_mask, "");

Please give meaningful messages to assertion failures.

David
-----

Vladimir Kozlov said the following on 08/03/10 15:45:
> I updated webrev:
> 
> http://cr.openjdk.java.net/~kvn/6973963/webrev.01
> 
> Additional changes:
> 
> 1. Used new option StressRecompilation instead of CompileTheWorld.
>    I am still not sure if it is OK since we never test our stress options.
> 
> 2. Added new allocation type: STACK_OR_EMBEDDED for cases when new() is 
> not called.
> 
> 3. Added ResourceObj destructor to zap _allocation field.
> 
> 4. Added assert into get_allocation_type() to check that 'this' address 
> is still encoded in _allocation.
>    Found several cases where it was not true, have to add copy 
> constructor and assignment operator.
> 
> 5. Moved all new methods with asserts into allocation.cpp.
> 
> 6. The added assert failed for CodeBuffer since it destroys itself 
> inside destructor before ResourceObj destructor called.
>    Moved Copy::fill_to_bytes(badResourceValue) into CodeBuffer::operator 
> delete().
> 
> 7. Replaced PhaseCFG::_node_latency field with pointer since it is valid 
> only inside resource mark in GlobalCodeMotion().
> 
> Thanks,
> Vladimir
> 
> Vladimir Kozlov wrote:
>> Thank you, Tom
>>
>> RESOURCE_AREA case combines 3 cases:
>>
>> GrowableArray<int> *foo = new GrowableArray<int>(size); // resource 
>> array allocation
>>
>> GrowableArray<int> foo; // stack allocation
>>
>> class A : public ResourceObj {
>>   GrowableArray<int> foo; // embedding allocation
>> }
>>
>> In all this cases GrowableArray::_data array is allocated in thread local
>> resource area. GrowableArray calls all these cases as stack allocation
>> and I followed this naming.
>>
>> But you are right I can separate 2 last cases into "stack" (stack or 
>> embedding)
>> allocation, I still have 1 bit in allocation_type :)
>>
>> Thanks,
>> Vladimir
>>
>> Tom Rodriguez wrote:
>>> So doesn't this triple the amount of time taken by CTW?  That seems 
>>> kind of extreme for the default mode.  Maybe it should be under a 
>>> flag?  The allocation.hpp changes look a little sketchy. +     if 
>>> (~(_allocation | allocation_mask) != (uintptr_t)this) {
>>> +       set_allocation_type((address)this, RESOURCE_AREA);
>>>
>>> Why is this case called RESOURCE_AREA?  Isn't this the stack or 
>>> embedding case?
>>>
>>> This doesn't make sense either.
>>>
>>> !   bool allocated_on_stack()     { return get_allocation_type() == 
>>> RESOURCE_AREA; }
>>>
>>> Anyway, the existing logic around this seemed sketchy so I can't 
>>> quite say whether this is better or not.  I'll have to leave that to 
>>> someone else.
>>>
>>> The GrowableArray changes themselves look fine.
>>>
>>> tom
>>>
>>> On Aug 2, 2010, at 3:01 PM, Vladimir Kozlov wrote:
>>>
>>>> Forwarding to GC and Runtime groups since it is common code.
>>>>
>>>> Vladimir
>>>>
>>>> -------- Original Message --------
>>>> Subject: Request for reviews (M): 6973963: SEGV in 
>>>> ciBlock::start_bci() with EA
>>>> Date: Mon, 02 Aug 2010 14:57:25 -0700
>>>> From: Vladimir Kozlov <vladimir.kozlov at oracle.com>
>>>> To: hotspot compiler <hotspot-compiler-dev at openjdk.java.net>
>>>>
>>>> http://cr.openjdk.java.net/~kvn/6973963/webrev
>>>>
>>>> Fixed 6973963: SEGV in ciBlock::start_bci() with EA
>>>>
>>>> I added stress recompilation during CompileTheWorld and found this 
>>>> case.
>>>> It is similar to 6968368. BCEscapeAnalyzer::do_analysis() calls
>>>> ciMethod::get_method_blocks() which calls constructor ciMethodBlocks.
>>>> This constructor allocates GrowableArray elements on stack (thread
>>>> local resource area). As result when the method recompiled without EA
>>>> _blocks->_data is NULL.
>>>>
>>>> Solution:
>>>> Added stress recompilation during CompileTheWorld: recompile with
>>>> subsume_loads = false and do_escape_analysis = false.
>>>> Added more checks into ResourceObj and growableArray to verify 
>>>> correctness
>>>> of allocation. I have to relax the new assert in GrowableArray when
>>>> elements are allocated on arena to allow allocattion of 
>>>> GrowableArray object
>>>> as a part of an other object (for example, in ConnectionGraph and 
>>>> SuperWord).
>>>>
>>>> Tested with failed cases, CTW.
>>>

From David.Holmes at oracle.com  Tue Aug  3 05:13:37 2010
From: David.Holmes at oracle.com (David Holmes)
Date: Tue, 03 Aug 2010 22:13:37 +1000
Subject: [Fwd: Request for reviews (M): 6973963: SEGV in
	ciBlock::start_bci() with EA]
In-Reply-To: <4C57E268.5060602@oracle.com>
References: <4C57401E.5040204@oracle.com>	<C3196616-FBD2-4282-8A63-ACD697F31781@oracle.com>	<4C574904.2050307@oracle.com>
	<4C57ACE6.6090702@oracle.com> <4C57B5D0.4090807@oracle.com>
	<4C57E268.5060602@oracle.com>
Message-ID: <4C5807F1.1070607@oracle.com>

Vladimir Kozlov said the following on 08/03/10 19:33:
> On 8/2/10 11:23 PM, David Holmes wrote:
>> src/share/vm/asm/codeBuffer.cpp
>>
>> + void CodeBuffer::operator delete(void* p) {
>> + ResourceObj::operator delete(p);
>> #ifdef ASSERT
>> ! Copy::fill_to_bytes(p, sizeof(CodeBuffer), badResourceValue);
>> #endif
>> }
>>
>> You moved the copy from the destructor to operator delete, but now you 
>> access p after you have deleted it.
> 
> Yes, you are right. But CodeBuffer is never allocated on C heap and 
> ResourceObj::delete(p)
> should be called only for allocation on C_HEAP. So this code should 
> never be executed,

I'm confused. CodeBuffer is a StackObj, not a ResourceObj, so why would 
you call ResourceObj::delete in the first place ??

> I will replace it with ShouldNotCallThis() in CodeBuffer::delete(void* p).
> Which leaves the original problem: I need to find how to zap CodeBuffer 
> after ResourceObj destructor
> which is called after CodeBuffer destructor.

Again I'm missing something - what is the connection between CodeBuffer 
and ResourceObj ?

> Note: the problem here is not CodeBuffer but its field: OopRecorder  
> _default_oop_recorder;

Explain that to me off-list if you like, I'm not familiar with this part 
of the code.

>> src/share/vm/memory/allocation.hpp
>>
>> ! ~ResourceObj();
>>
>> You've added a destructor, but subclasses define their own destructors 
>> - so doesn't this need to be virtual?
> 
> Constructor and destructor are special methods. Super class's default 
> destructor are always called from
> subclass's destructor. The only case when you need to declare it as 
> virtual if subclass object is assigned to
> local/field of super class type and compiler does not know what the 
> original subclass was.
> 
> ResourceObj *f = new OopRecorder();
> ...
> delete f;
> 
> We don't use such constructions.

Ok. Just wanted to check. In another codebase we got bitten by a cleanup 
method in the superclass that did "delete this;" which did not invoke 
the destructor for the dynamic type of 'this' because the destructor was 
not virtual.

Cheers,
David

From John.Coomes at oracle.com  Tue Aug  3 14:16:22 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Tue, 3 Aug 2010 14:16:22 -0700
Subject: review request (S): 6966222: G1: simplify TaskQueue overflow
	handling
In-Reply-To: <19502.20117.466404.455682@oracle.com>
References: <19502.20117.466404.455682@oracle.com>
Message-ID: <19544.34598.254904.376459@oracle.com>

I (John.Coomes at oracle.com) wrote:
> I'd appreciate reviews of the changes to simplify TaskQueue overflow
> handling in G1.
> 
> 	http://cr.openjdk.java.net/~jcoomes/6966222-taskqueue-g1/

I updated the webrev; the main difference is to format the output as a
table.  See the webrev comments for more details.

I'd particularly like people to verify that the change in the duration
that the stats_lock() is held is correct (g1CollectedHeap.cpp).

> The changes are limited to 2 files, and more lines were deleted than
> added :-).

Unfortunately, the above two statements are no longer accurate :-(.

-John


From John.Coomes at oracle.com  Tue Aug  3 14:20:51 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Tue, 3 Aug 2010 14:20:51 -0700
Subject: review request (S+):  6970376 ParNew shared TaskQueue stats
Message-ID: <19544.34867.434755.823697@oracle.com>

Continuing in my taskqueue series, I'd appreciate reviews of the
changes to ParNew to use the built-in taskqueue stats:

http://cr.openjdk.java.net/~jcoomes/6970376-taskqueue-parnew/

-John


From y.s.ramakrishna at oracle.com  Tue Aug  3 16:28:30 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Tue, 03 Aug 2010 16:28:30 -0700
Subject: review request (S+):  6970376 ParNew shared TaskQueue stats
In-Reply-To: <19544.34867.434755.823697@oracle.com>
References: <19544.34867.434755.823697@oracle.com>
Message-ID: <4C58A61E.4000505@oracle.com>

Looks good to me.

-- ramki

John Coomes wrote:
> Continuing in my taskqueue series, I'd appreciate reviews of the
> changes to ParNew to use the built-in taskqueue stats:
> 
> http://cr.openjdk.java.net/~jcoomes/6970376-taskqueue-parnew/
> 
> -John
> 


From y.s.ramakrishna at oracle.com  Tue Aug  3 16:48:51 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Tue, 03 Aug 2010 16:48:51 -0700
Subject: review request (S): 6966222: G1: simplify TaskQueue overflow
	handling
In-Reply-To: <19544.34598.254904.376459@oracle.com>
References: <19502.20117.466404.455682@oracle.com>
	<19544.34598.254904.376459@oracle.com>
Message-ID: <4C58AAE3.8000207@oracle.com>

Also looks good.

-- ramki

John Coomes wrote:
> I (John.Coomes at oracle.com) wrote:
>> I'd appreciate reviews of the changes to simplify TaskQueue overflow
>> handling in G1.
>>
>> 	http://cr.openjdk.java.net/~jcoomes/6966222-taskqueue-g1/
> 
> I updated the webrev; the main difference is to format the output as a
> table.  See the webrev comments for more details.
> 
> I'd particularly like people to verify that the change in the duration
> that the stats_lock() is held is correct (g1CollectedHeap.cpp).
> 
>> The changes are limited to 2 files, and more lines were deleted than
>> added :-).
> 
> Unfortunately, the above two statements are no longer accurate :-(.
> 
> -John
> 


From John.Coomes at oracle.com  Tue Aug  3 17:04:50 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Tue, 3 Aug 2010 17:04:50 -0700
Subject: review request (S+):  6970376 ParNew shared TaskQueue stats
In-Reply-To: <4C58A61E.4000505@oracle.com>
References: <19544.34867.434755.823697@oracle.com>
	<4C58A61E.4000505@oracle.com>
Message-ID: <19544.44706.906394.605260@oracle.com>

Y. Srinivas Ramakrishna (y.s.ramakrishna at oracle.com) wrote:
> Looks good to me.

That was quick :-).  Thanks!

-John

> John Coomes wrote:
> > Continuing in my taskqueue series, I'd appreciate reviews of the
> > changes to ParNew to use the built-in taskqueue stats:
> > 
> > http://cr.openjdk.java.net/~jcoomes/6970376-taskqueue-parnew/
> > 
> > -John
> > 
> 


From John.Coomes at oracle.com  Tue Aug  3 17:11:37 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Tue, 3 Aug 2010 17:11:37 -0700
Subject: review request (S): 6966222: G1: simplify TaskQueue overflow
	handling
In-Reply-To: <4C58AAE3.8000207@oracle.com>
References: <19502.20117.466404.455682@oracle.com>
	<19544.34598.254904.376459@oracle.com>
	<4C58AAE3.8000207@oracle.com>
Message-ID: <19544.45113.487582.477345@oracle.com>

Y. Srinivas Ramakrishna (y.s.ramakrishna at oracle.com) wrote:
> Also looks good.

Wow, you're on a roll--maybe I should get a few more things out for
review :-).

Many thanks.

-John

> John Coomes wrote:
> > I (John.Coomes at oracle.com) wrote:
> >> I'd appreciate reviews of the changes to simplify TaskQueue overflow
> >> handling in G1.
> >>
> >> 	http://cr.openjdk.java.net/~jcoomes/6966222-taskqueue-g1/
> > 
> > I updated the webrev; the main difference is to format the output as a
> > table.  See the webrev comments for more details.
> > 
> > I'd particularly like people to verify that the change in the duration
> > that the stats_lock() is held is correct (g1CollectedHeap.cpp).
> > 
> >> The changes are limited to 2 files, and more lines were deleted than
> >> added :-).
> > 
> > Unfortunately, the above two statements are no longer accurate :-(.
> > 
> > -John
> > 
> 


From tony.printezis at oracle.com  Thu Aug  5 08:09:54 2010
From: tony.printezis at oracle.com (tony.printezis at oracle.com)
Date: Thu, 05 Aug 2010 15:09:54 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 2 new changesets
Message-ID: <20100805151006.EAECE47F24@hg.openjdk.java.net>

Changeset: 9d7a8ab3736b
Author:    tonyp
Date:      2010-07-22 10:27 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/9d7a8ab3736b

6962589: remove breadth first scanning code from parallel gc
Summary: Remove the breadth-first copying order from ParallelScavenge and use depth-first by default.
Reviewed-by: jcoomes, ysr, johnc

! src/share/vm/gc_implementation/includeDB_gc_parallelScavenge
! src/share/vm/gc_implementation/parallelScavenge/cardTableExtension.cpp
- src/share/vm/gc_implementation/parallelScavenge/prefetchQueue.hpp
! src/share/vm/gc_implementation/parallelScavenge/psPromotionManager.cpp
! src/share/vm/gc_implementation/parallelScavenge/psPromotionManager.hpp
! src/share/vm/gc_implementation/parallelScavenge/psPromotionManager.inline.hpp
! src/share/vm/gc_implementation/parallelScavenge/psScavenge.cpp
! src/share/vm/gc_implementation/parallelScavenge/psScavenge.inline.hpp
! src/share/vm/gc_implementation/parallelScavenge/psTasks.cpp
! src/share/vm/oops/arrayKlassKlass.cpp
! src/share/vm/oops/compiledICHolderKlass.cpp
! src/share/vm/oops/constMethodKlass.cpp
! src/share/vm/oops/constantPoolKlass.cpp
! src/share/vm/oops/cpCacheKlass.cpp
! src/share/vm/oops/instanceKlass.cpp
! src/share/vm/oops/instanceKlass.hpp
! src/share/vm/oops/instanceKlassKlass.cpp
! src/share/vm/oops/instanceRefKlass.cpp
! src/share/vm/oops/klassKlass.cpp
! src/share/vm/oops/klassPS.hpp
! src/share/vm/oops/methodDataKlass.cpp
! src/share/vm/oops/methodKlass.cpp
! src/share/vm/oops/objArrayKlass.cpp
! src/share/vm/oops/objArrayKlassKlass.cpp
! src/share/vm/oops/oop.hpp
! src/share/vm/oops/oop.psgc.inline.hpp
! src/share/vm/oops/symbolKlass.cpp
! src/share/vm/oops/typeArrayKlass.cpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/globals.hpp

Changeset: 0ce1569c90e5
Author:    tonyp
Date:      2010-08-04 13:03 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0ce1569c90e5

6963209: G1: remove the concept of abandoned pauses
Summary: As part of 6944166 we disabled the concept of abandoned pauses (i.e., if the collection set is empty, we would still try to do a pause even if it is to update the RSets and scan the roots). This changeset removes the code and structures associated with abandoned pauses.
Reviewed-by: iveresov, johnc

! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp
! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp
! src/share/vm/gc_implementation/g1/g1CollectorPolicy.hpp


From vladimir.kozlov at oracle.com  Thu Aug  5 15:59:48 2010
From: vladimir.kozlov at oracle.com (Vladimir Kozlov)
Date: Thu, 05 Aug 2010 15:59:48 -0700
Subject: Request for reviews (S): 6975078: assert(allocated_on_res_area()
	|| allocated_on_C_heap() || allocated_on_arena()
Message-ID: <4C5B4264.2020004@oracle.com>

Sending also to GC since I touched G1 code :)

http://cr.openjdk.java.net/~kvn/6975078/webrev

Fixed 6975078: assert(allocated_on_res_area() || allocated_on_C_heap() || allocated_on_arena()

The assert is from my fix for 6973963 and I can't reproduce this failure.
void emit_call_reloc() {
   MacroAssembler _masm(&cbuf); <<< asserts here, allocation on stack.

It could be because ~ResourceObj() destructor is not called for
_masm but I doubt it. In 6973963 changes to track correctness of
allocation type (to separate it from garbage on stack) I encoded
(negated) 'this' address into _allocation value and zap it in
~ResourceObj() destructor.

Most likely it is because the garbage value on stack is equal to
~(address of _masm on stack). For example, for 0xffffffff613fb4d0
(sp from hs_err file) it could be 0x9ec04b20.
I thought it would be impossible but I was wrong, it seems.

Solution:
Pass the check in constructor ResourceObj() if _allocation has
a value which looks like an allocation on stack and it is really
allocated on stack.

I also did cleanup:
- added 'const' to ResourceObj access methods,
- fixed few typos and comments,
- replaced in G1 call to ResourceObj::new() with ResourceObj::set_allocation_type().

JPRT.

From john.cuthbertson at sun.com  Fri Aug  6 13:46:54 2010
From: john.cuthbertson at sun.com (john.cuthbertson at sun.com)
Date: Fri, 06 Aug 2010 20:46:54 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6930581: G1: assert(ParallelGCThreads >
	1 || n_yielded() == _hrrs->occupied(), "Should have yielded all the ..
Message-ID: <20100806204659.59A5947F64@hg.openjdk.java.net>

Changeset: a03ae377b2e8
Author:    johnc
Date:      2010-08-06 10:17 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a03ae377b2e8

6930581: G1: assert(ParallelGCThreads > 1 || n_yielded() == _hrrs->occupied(),"Should have yielded all the ..
Summary: During RSet updating, when ParallelGCThreads is zero, references that point into the collection set are added directly the referenced region's RSet. This can cause the sparse table in the RSet to expand. RSet scanning and the "occupied" routine will then operate on different instances of the sparse table causing the assert to trip. This may also cause some cards added post expansion to be missed during RSet scanning. When ParallelGCThreads is non-zero such references are recorded on the "references to be scanned" queue and the card containing the reference is recorded in a dirty card queue for use in the event of an evacuation failure. Employ the parallel code in the serial case to avoid expanding the RSets of regions in the collection set.
Reviewed-by: iveresov, ysr, tonyp

! src/share/vm/gc_implementation/g1/g1RemSet.cpp
! src/share/vm/gc_implementation/g1/g1RemSet.hpp
! src/share/vm/gc_implementation/g1/g1RemSet.inline.hpp
! src/share/vm/gc_implementation/g1/sparsePRT.cpp


From john.coomes at oracle.com  Mon Aug  9 08:29:44 2010
From: john.coomes at oracle.com (john.coomes at oracle.com)
Date: Mon, 09 Aug 2010 15:29:44 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6966222: G1: simplify TaskQueue overflow
	handling
Message-ID: <20100809152948.9192947002@hg.openjdk.java.net>

Changeset: 5f429ee79634
Author:    jcoomes
Date:      2010-08-09 05:41 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5f429ee79634

6966222: G1: simplify TaskQueue overflow handling
Reviewed-by: tonyp, ysr

! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.inline.hpp
! src/share/vm/utilities/taskqueue.cpp
! src/share/vm/utilities/taskqueue.hpp


From john.coomes at oracle.com  Tue Aug 10 09:57:13 2010
From: john.coomes at oracle.com (john.coomes at oracle.com)
Date: Tue, 10 Aug 2010 16:57:13 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6970376: ParNew: shared TaskQueue
	statistics
Message-ID: <20100810165724.1021E47058@hg.openjdk.java.net>

Changeset: 94251661de76
Author:    jcoomes
Date:      2010-08-09 18:03 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/94251661de76

6970376: ParNew: shared TaskQueue statistics
Reviewed-by: ysr

! src/share/vm/gc_implementation/parNew/parNewGeneration.cpp
! src/share/vm/gc_implementation/parNew/parNewGeneration.hpp


From y.s.ramakrishna at oracle.com  Tue Aug 10 23:16:53 2010
From: y.s.ramakrishna at oracle.com (y.s.ramakrishna at oracle.com)
Date: Wed, 11 Aug 2010 06:16:53 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6973570: OrderAccess::storestore()
	scales poorly on multi-socket x64 and sparc: cache-line ping-ponging
Message-ID: <20100811061656.23ABC4708B@hg.openjdk.java.net>

Changeset: a6bff45449bc
Author:    ysr
Date:      2010-08-10 14:53 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a6bff45449bc

6973570: OrderAccess::storestore() scales poorly on multi-socket x64 and sparc: cache-line ping-ponging
Summary: volatile store to static variable removed in favour of a volatile store to stack to avoid excessive cache coherency traffic; verified that the volatile store is not elided by any of our current compilers.
Reviewed-by: dholmes, dice, jcoomes, kvn

! src/os_cpu/linux_sparc/vm/orderAccess_linux_sparc.inline.hpp
! src/os_cpu/linux_x86/vm/orderAccess_linux_x86.inline.hpp
! src/os_cpu/solaris_sparc/vm/orderAccess_solaris_sparc.inline.hpp
! src/os_cpu/solaris_x86/vm/orderAccess_solaris_x86.inline.hpp
! src/os_cpu/windows_x86/vm/orderAccess_windows_x86.inline.hpp
! src/share/vm/runtime/orderAccess.cpp
! src/share/vm/runtime/orderAccess.hpp


From John.Coomes at oracle.com  Wed Aug 11 12:20:50 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Wed, 11 Aug 2010 12:20:50 -0700
Subject: review request (XXS): 6976378 stats always printed in debug builds
Message-ID: <19554.63506.931990.754198@oracle.com>

I unintentionally left stats printing enabled unconditionally in debug
builds with ParNew (oops).  This change guards stats printing with
PrintGCDetails && ParallelGCVerbose.

	http://cr.openjdk.java.net/~jcoomes/6976378-taskqueue-parnew/

Thanks for any feedback.

-John


From john.coomes at oracle.com  Wed Aug 11 16:06:44 2010
From: john.coomes at oracle.com (john.coomes at oracle.com)
Date: Wed, 11 Aug 2010 23:06:44 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6976378: ParNew: stats are printed
	unconditionally in debug builds
Message-ID: <20100811230646.450F3470B8@hg.openjdk.java.net>

Changeset: 2d6b74c9a797
Author:    jcoomes
Date:      2010-08-11 13:12 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/2d6b74c9a797

6976378: ParNew:  stats are printed unconditionally in debug builds
Reviewed-by: tonyp

! src/share/vm/gc_implementation/parNew/parNewGeneration.cpp


From john.cuthbertson at sun.com  Sat Aug 14 05:22:22 2010
From: john.cuthbertson at sun.com (john.cuthbertson at sun.com)
Date: Sat, 14 Aug 2010 12:22:22 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 22 new changesets
Message-ID: <20100814122303.4AAAF4719A@hg.openjdk.java.net>

Changeset: a81afd9c293c
Author:    alanb
Date:      2010-07-16 13:14 +0100
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a81afd9c293c

6649594: Intermittent IOExceptions during dynamic attach on linux and solaris
Reviewed-by: dcubed, dholmes

! src/os/linux/vm/attachListener_linux.cpp
! src/os/solaris/vm/attachListener_solaris.cpp

Changeset: 920aa833fd16
Author:    apangin
Date:      2010-07-17 21:49 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/920aa833fd16

Merge


Changeset: a5c9d63a187d
Author:    apangin
Date:      2010-07-20 08:41 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a5c9d63a187d

6964170: Verifier crashes
Summary: Check if klassOop != NULL rather than klass_part != NULL
Reviewed-by: kamg, never

! src/share/vm/classfile/verificationType.cpp
! src/share/vm/classfile/verifier.cpp

Changeset: 7f0fdccac34f
Author:    apangin
Date:      2010-07-25 07:31 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/7f0fdccac34f

Merge

! src/share/vm/classfile/verifier.cpp

Changeset: 3d90023429ec
Author:    aph
Date:      2010-07-28 17:38 +0100
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/3d90023429ec

6888526: Linux getCurrentThreadCpuTime is drastically slower than Windows
Reviewed-by: dcubed, dholmes

! src/os/linux/vm/globals_linux.hpp
! src/share/vm/runtime/arguments.cpp

Changeset: a64438a2b7e8
Author:    coleenp
Date:      2010-07-28 17:57 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a64438a2b7e8

6958465: Sparc aten build24.0: openjdk-7.ea-b96 failed Error: Formal argument ... requires an lvalue
Summary: Fix compilation errors.  Made non-const references const so can be assigned with lvalue.
Reviewed-by: phh, xlu

! src/cpu/sparc/vm/assembler_sparc.cpp
! src/cpu/sparc/vm/assembler_sparc.hpp
! src/cpu/sparc/vm/assembler_sparc.inline.hpp

Changeset: 126ea7725993
Author:    bobv
Date:      2010-08-03 08:13 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/126ea7725993

6953477: Increase portability and flexibility of building Hotspot
Summary: A collection of portability improvements including shared code support for PPC, ARM platforms, software floating point, cross compilation support and improvements in error crash detail.
Reviewed-by: phh, never, coleenp, dholmes

! agent/src/os/linux/ps_proc.c
! make/Makefile
! make/defs.make
! make/linux/makefiles/build_vm_def.sh
! make/linux/makefiles/buildtree.make
! make/linux/makefiles/defs.make
! make/linux/makefiles/gcc.make
! make/linux/makefiles/product.make
! make/linux/makefiles/sa.make
! make/linux/makefiles/saproc.make
! make/linux/makefiles/vm.make
! make/solaris/makefiles/defs.make
! src/cpu/sparc/vm/bytecodeInterpreter_sparc.inline.hpp
! src/cpu/sparc/vm/c1_LIRGenerator_sparc.cpp
! src/cpu/sparc/vm/c1_Runtime1_sparc.cpp
! src/cpu/sparc/vm/interpreterRT_sparc.cpp
! src/cpu/sparc/vm/javaFrameAnchor_sparc.hpp
! src/cpu/sparc/vm/templateTable_sparc.cpp
! src/cpu/x86/vm/bytecodeInterpreter_x86.inline.hpp
! src/cpu/x86/vm/c1_LIRGenerator_x86.cpp
! src/cpu/x86/vm/c1_Runtime1_x86.cpp
! src/cpu/x86/vm/frame_x86.cpp
! src/cpu/x86/vm/interpreterRT_x86_32.cpp
! src/cpu/x86/vm/javaFrameAnchor_x86.hpp
! src/cpu/x86/vm/templateTable_x86_32.cpp
! src/cpu/x86/vm/templateTable_x86_64.cpp
! src/os/linux/launcher/java_md.c
! src/os/linux/vm/os_linux.cpp
! src/os/solaris/vm/os_solaris.cpp
! src/os/windows/vm/os_windows.cpp
! src/os_cpu/linux_sparc/vm/thread_linux_sparc.cpp
! src/os_cpu/linux_x86/vm/os_linux_x86.cpp
! src/os_cpu/linux_x86/vm/thread_linux_x86.cpp
! src/os_cpu/linux_zero/vm/thread_linux_zero.cpp
! src/os_cpu/solaris_sparc/vm/os_solaris_sparc.cpp
! src/os_cpu/solaris_sparc/vm/thread_solaris_sparc.cpp
! src/os_cpu/solaris_x86/vm/os_solaris_x86.cpp
! src/os_cpu/solaris_x86/vm/thread_solaris_x86.cpp
! src/os_cpu/windows_x86/vm/os_windows_x86.cpp
! src/os_cpu/windows_x86/vm/thread_windows_x86.cpp
! src/share/vm/asm/codeBuffer.hpp
! src/share/vm/c1/c1_CodeStubs.hpp
! src/share/vm/c1/c1_Compilation.hpp
! src/share/vm/c1/c1_FrameMap.cpp
! src/share/vm/c1/c1_FrameMap.hpp
! src/share/vm/c1/c1_LIR.cpp
! src/share/vm/c1/c1_LIR.hpp
! src/share/vm/c1/c1_LIRGenerator.cpp
! src/share/vm/c1/c1_LIRGenerator.hpp
! src/share/vm/c1/c1_LinearScan.cpp
! src/share/vm/c1/c1_Runtime1.cpp
! src/share/vm/c1/c1_Runtime1.hpp
! src/share/vm/code/codeBlob.cpp
! src/share/vm/code/codeBlob.hpp
! src/share/vm/code/nmethod.hpp
! src/share/vm/code/vtableStubs.cpp
! src/share/vm/code/vtableStubs.hpp
! src/share/vm/compiler/disassembler.cpp
! src/share/vm/includeDB_compiler1
! src/share/vm/includeDB_core
! src/share/vm/interpreter/bytecodeInterpreter.cpp
! src/share/vm/interpreter/bytecodeInterpreter.hpp
! src/share/vm/interpreter/bytecodeInterpreter.inline.hpp
! src/share/vm/interpreter/interpreter.cpp
! src/share/vm/interpreter/interpreter.hpp
! src/share/vm/interpreter/oopMapCache.cpp
! src/share/vm/memory/allocation.cpp
! src/share/vm/memory/allocation.hpp
! src/share/vm/memory/genCollectedHeap.cpp
! src/share/vm/memory/generation.hpp
! src/share/vm/oops/arrayKlass.cpp
! src/share/vm/oops/arrayKlass.hpp
! src/share/vm/oops/arrayKlassKlass.cpp
! src/share/vm/oops/arrayKlassKlass.hpp
! src/share/vm/oops/compiledICHolderKlass.cpp
! src/share/vm/oops/compiledICHolderKlass.hpp
! src/share/vm/oops/constMethodKlass.cpp
! src/share/vm/oops/constMethodKlass.hpp
! src/share/vm/oops/constantPoolKlass.cpp
! src/share/vm/oops/constantPoolKlass.hpp
! src/share/vm/oops/cpCacheKlass.cpp
! src/share/vm/oops/cpCacheKlass.hpp
! src/share/vm/oops/generateOopMap.cpp
! src/share/vm/oops/klass.cpp
! src/share/vm/oops/klass.hpp
! src/share/vm/oops/klassKlass.cpp
! src/share/vm/oops/klassKlass.hpp
! src/share/vm/oops/oop.cpp
! src/share/vm/prims/jni.cpp
! src/share/vm/prims/jvmtiEnvThreadState.hpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/frame.cpp
! src/share/vm/runtime/frame.hpp
! src/share/vm/runtime/globals.hpp
! src/share/vm/runtime/java.cpp
! src/share/vm/runtime/javaFrameAnchor.hpp
! src/share/vm/runtime/os.cpp
! src/share/vm/runtime/os.hpp
! src/share/vm/runtime/sharedRuntime.cpp
! src/share/vm/runtime/sharedRuntime.hpp
! src/share/vm/runtime/sharedRuntimeTrans.cpp
! src/share/vm/runtime/signature.hpp
! src/share/vm/runtime/stubCodeGenerator.cpp
! src/share/vm/runtime/stubCodeGenerator.hpp
! src/share/vm/runtime/thread.cpp
! src/share/vm/runtime/thread.hpp
! src/share/vm/runtime/vm_version.cpp
! src/share/vm/runtime/vm_version.hpp
! src/share/vm/utilities/debug.cpp
! src/share/vm/utilities/globalDefinitions_gcc.hpp
! src/share/vm/utilities/macros.hpp
! src/share/vm/utilities/vmError.cpp
! src/share/vm/utilities/vmError.hpp

Changeset: e5dfb3ccb88b
Author:    kvn
Date:      2010-07-23 10:07 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/e5dfb3ccb88b

6969569: assert(is_static() && is_constant()) failed: illegal call to constant_value()
Summary: Add missing is_static guard.
Reviewed-by: twisti

! src/share/vm/ci/ciField.cpp
! src/share/vm/opto/macro.cpp

Changeset: 99ceb0e99c9e
Author:    never
Date:      2010-07-26 15:58 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/99ceb0e99c9e

Merge


Changeset: 66c5dadb4d61
Author:    kvn
Date:      2010-07-30 10:21 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/66c5dadb4d61

6973308: Missing zero length check before repne scas in check_klass_subtype_slow_path()
Summary: set Z = 0 (not equal) before repne_scan() to indicate that class was not found when RCX == 0.
Reviewed-by: never, phh

! src/cpu/x86/vm/assembler_x86.cpp
! src/cpu/x86/vm/assembler_x86.hpp
! src/share/vm/runtime/globals.hpp
! src/share/vm/utilities/exceptions.cpp
! src/share/vm/utilities/exceptions.hpp
! src/share/vm/utilities/vmError.cpp

Changeset: 0e35fa8ebccd
Author:    kvn
Date:      2010-08-03 15:55 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0e35fa8ebccd

6973963: SEGV in ciBlock::start_bci() with EA
Summary: Added more checks into ResourceObj and growableArray to verify correctness of allocation type.
Reviewed-by: never, coleenp, dholmes

! src/share/vm/asm/codeBuffer.cpp
! src/share/vm/asm/codeBuffer.hpp
! src/share/vm/ci/ciInstanceKlass.cpp
! src/share/vm/ci/ciMethodBlocks.cpp
! src/share/vm/ci/ciTypeFlow.cpp
! src/share/vm/classfile/classFileParser.cpp
! src/share/vm/memory/allocation.cpp
! src/share/vm/memory/allocation.hpp
! src/share/vm/opto/block.cpp
! src/share/vm/opto/block.hpp
! src/share/vm/opto/c2_globals.hpp
! src/share/vm/opto/c2compiler.cpp
! src/share/vm/opto/chaitin.cpp
! src/share/vm/opto/compile.cpp
! src/share/vm/opto/gcm.cpp
! src/share/vm/opto/lcm.cpp
! src/share/vm/utilities/growableArray.hpp

Changeset: 0e09207fc81b
Author:    kvn
Date:      2010-08-04 17:42 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0e09207fc81b

6974682: CTW: assert(target != NULL) failed: must not be null
Summary: Add address table size to constant section size.
Reviewed-by: never

! src/share/vm/opto/output.cpp

Changeset: fb8abd207dbe
Author:    kvn
Date:      2010-08-06 11:53 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/fb8abd207dbe

6975049: nsk/regression/b4287029 crashes with -Xss64 on solaris-i586
Summary: Tell C++ to not inline so much by using flag -xspace.
Reviewed-by: ysr

! make/solaris/makefiles/sparcWorks.make

Changeset: 2dfd013a7465
Author:    kvn
Date:      2010-08-09 15:17 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/2dfd013a7465

6975078: assert(allocated_on_res_area() || allocated_on_C_heap() || allocated_on_arena()
Summary: Pass the check in ResourceObj() if _allocation value is already set and object is allocated on stack.
Reviewed-by: dholmes, johnc

! src/share/vm/gc_implementation/g1/collectionSetChooser.cpp
! src/share/vm/gc_implementation/g1/heapRegionSeq.cpp
! src/share/vm/memory/allocation.cpp
! src/share/vm/memory/allocation.hpp
! src/share/vm/runtime/thread.cpp
! src/share/vm/runtime/thread.hpp

Changeset: f4f596978298
Author:    never
Date:      2010-08-09 17:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f4f596978298

Merge

! src/share/vm/asm/codeBuffer.hpp
! src/share/vm/memory/allocation.cpp
! src/share/vm/memory/allocation.hpp
! src/share/vm/runtime/globals.hpp
! src/share/vm/runtime/thread.cpp
! src/share/vm/runtime/thread.hpp
! src/share/vm/utilities/vmError.cpp

Changeset: 36519c19beeb
Author:    never
Date:      2010-08-10 12:15 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/36519c19beeb

6975027: use of movptr to set length of array
Reviewed-by: kvn, iveresov

! src/cpu/x86/vm/assembler_x86.cpp

Changeset: 4a665be40fd3
Author:    twisti
Date:      2010-08-11 01:17 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/4a665be40fd3

6975855: don't emit deopt MH handler in C1 if not required
Summary: This CR implements the same for C1 as 6926782 for C2.
Reviewed-by: never

! src/share/vm/c1/c1_Compilation.cpp
! src/share/vm/c1/c1_Compilation.hpp
! src/share/vm/c1/c1_LIRAssembler.cpp
! src/share/vm/code/nmethod.cpp

Changeset: d2ede61b7a12
Author:    twisti
Date:      2010-08-11 05:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/d2ede61b7a12

6976186: integrate Shark HotSpot changes
Summary: Shark is a JIT compiler for Zero that uses the LLVM compiler infrastructure.
Reviewed-by: kvn, twisti
Contributed-by: Gary Benson <gbenson at redhat.com>

! make/Makefile
! make/linux/Makefile
! make/linux/makefiles/gcc.make
+ make/linux/makefiles/shark.make
! make/linux/makefiles/top.make
! make/linux/makefiles/vm.make
! src/cpu/zero/vm/disassembler_zero.hpp
+ src/cpu/zero/vm/shark_globals_zero.hpp
! src/share/vm/ci/ciMethod.cpp
! src/share/vm/ci/ciMethod.hpp
! src/share/vm/code/nmethod.cpp
! src/share/vm/code/nmethod.hpp
! src/share/vm/compiler/abstractCompiler.hpp
! src/share/vm/compiler/compileBroker.cpp
! src/share/vm/compiler/disassembler.cpp
+ src/share/vm/includeDB_shark
! src/share/vm/memory/cardTableModRefBS.hpp
! src/share/vm/oops/methodOop.cpp
! src/share/vm/runtime/deoptimization.cpp
! src/share/vm/runtime/frame.cpp
! src/share/vm/runtime/globals.cpp
! src/share/vm/runtime/globals.hpp
! src/share/vm/runtime/vm_version.cpp
+ src/share/vm/shark/llvmHeaders.hpp
+ src/share/vm/shark/llvmValue.hpp
+ src/share/vm/shark/sharkBlock.cpp
+ src/share/vm/shark/sharkBlock.hpp
+ src/share/vm/shark/sharkBuilder.cpp
+ src/share/vm/shark/sharkBuilder.hpp
+ src/share/vm/shark/sharkCacheDecache.cpp
+ src/share/vm/shark/sharkCacheDecache.hpp
+ src/share/vm/shark/sharkCodeBuffer.hpp
+ src/share/vm/shark/sharkCompiler.cpp
+ src/share/vm/shark/sharkCompiler.hpp
+ src/share/vm/shark/sharkConstant.cpp
+ src/share/vm/shark/sharkConstant.hpp
+ src/share/vm/shark/sharkContext.cpp
+ src/share/vm/shark/sharkContext.hpp
+ src/share/vm/shark/sharkEntry.hpp
+ src/share/vm/shark/sharkFunction.cpp
+ src/share/vm/shark/sharkFunction.hpp
+ src/share/vm/shark/sharkInliner.cpp
+ src/share/vm/shark/sharkInliner.hpp
+ src/share/vm/shark/sharkIntrinsics.cpp
+ src/share/vm/shark/sharkIntrinsics.hpp
+ src/share/vm/shark/sharkInvariants.cpp
+ src/share/vm/shark/sharkInvariants.hpp
+ src/share/vm/shark/sharkMemoryManager.cpp
+ src/share/vm/shark/sharkMemoryManager.hpp
+ src/share/vm/shark/sharkNativeWrapper.cpp
+ src/share/vm/shark/sharkNativeWrapper.hpp
+ src/share/vm/shark/sharkRuntime.cpp
+ src/share/vm/shark/sharkRuntime.hpp
+ src/share/vm/shark/sharkStack.cpp
+ src/share/vm/shark/sharkStack.hpp
+ src/share/vm/shark/sharkState.cpp
+ src/share/vm/shark/sharkState.hpp
+ src/share/vm/shark/sharkStateScanner.cpp
+ src/share/vm/shark/sharkStateScanner.hpp
+ src/share/vm/shark/sharkTopLevelBlock.cpp
+ src/share/vm/shark/sharkTopLevelBlock.hpp
+ src/share/vm/shark/sharkType.hpp
+ src/share/vm/shark/sharkValue.cpp
+ src/share/vm/shark/sharkValue.hpp
+ src/share/vm/shark/shark_globals.cpp
+ src/share/vm/shark/shark_globals.hpp
! src/share/vm/utilities/macros.hpp

Changeset: 6c9cc03d8726
Author:    kvn
Date:      2010-08-11 10:48 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6c9cc03d8726

6973329: C2 with Zero based COOP produces code with broken anti-dependency on x86
Summary: Recompile without subsuming loads if RA try to clone a node with anti_dependence.
Reviewed-by: never

! src/share/vm/includeDB_compiler2
! src/share/vm/opto/lcm.cpp
! src/share/vm/opto/reg_split.cpp
+ test/compiler/6973329/Test.java

Changeset: ab3fd720516c
Author:    rasbold
Date:      2010-08-10 19:17 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/ab3fd720516c

6378314: Bad warning message when agent library not found. local directory is not searched.
Summary: Print a more detailed error message for agent library load failure.
Reviewed-by: jcoomes, never, ohair, coleenp
Contributed-by: jeremymanson at google.com

! src/share/vm/runtime/thread.cpp

Changeset: 21e519b91576
Author:    dcubed
Date:      2010-08-13 07:33 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/21e519b91576

Merge

! src/share/vm/runtime/thread.cpp

Changeset: 7fcd5f39bd7a
Author:    johnc
Date:      2010-08-14 00:47 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/7fcd5f39bd7a

Merge

- src/share/vm/gc_implementation/parallelScavenge/prefetchQueue.hpp
! src/share/vm/oops/arrayKlassKlass.cpp
! src/share/vm/oops/compiledICHolderKlass.cpp
! src/share/vm/oops/constMethodKlass.cpp
! src/share/vm/oops/constantPoolKlass.cpp
! src/share/vm/oops/cpCacheKlass.cpp
! src/share/vm/oops/klassKlass.cpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/globals.hpp


From David.Holmes at oracle.com  Fri Aug 13 16:23:10 2010
From: David.Holmes at oracle.com (David Holmes)
Date: Sat, 14 Aug 2010 09:23:10 +1000
Subject: write barrier and card marking
In-Reply-To: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
References: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
Message-ID: <4C65D3DE.5050803@oracle.com>

Tony,

This is a GC question not a runtime question so I've cc'ed the GC list 
and bcc'ed the runtime list.

David

Tony Guan said the following on 08/14/10 02:28:
> Hi there,
> 
> I wrote one collector of my own in hotspot, but now I have a problem.
> In the new collector, I inserted another generation between the
> defNewGeneration and TenuredGeneration. To collect this new
> generation, I modified the defNewGeneration collector, which is gc by
> copying survivors.
> 
> After the collection on this generation, I am now having an Null
> pointer exception in the java program. My guess is that the collector
> failed to identify the live objects in the generation. Here I need
> some one to tell me if there is any missed modification on the write
> barrier or card marking.
> 
> As far as I know, the write barrier works whenever a field write
> happens, without regards to which generation is influenced. So this
> means that the card of the written field will be marked. So my problem
> should be in the closures that check if there are any live objects in
> the generation. Am I right?
> 
> Or other than the card scanning, is there anything special that I
> should process?
> 
> Thanks a lot!
> 
> 
> Tony (Xiaohua Guan)

From y.s.ramakrishna at oracle.com  Sat Aug 14 11:25:39 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Sat, 14 Aug 2010 11:25:39 -0700
Subject: write barrier and card marking
In-Reply-To: <4C65D3DE.5050803@oracle.com>
References: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
	<4C65D3DE.5050803@oracle.com>
Message-ID: <4C66DFA3.8060902@oracle.com>

Tony,

Do you collect each generation independently, or
does a generation at level n (with youngest == 0)
collect all younger generations (i.e. level m < n)?

The usual failure pattern for missed remembered set entries
is that there is a "stale" pointer in a generation not collected
but with a pointer into a generation that was collected.
You can gainfully use the "mangling" feature in debug builds to
identify such cases sometimes. I have not generally seen null pointer
exceptions in such cases though (because that would mean that you are
getting hold of a null reference, which requires more work, i.e. clearing,
than forgetting to update an old reference to point to a new location),
just "broken" references to stale objects (these usually end up pointing
into the middle of unallocated space or into the middle of another object --
i.e. a location that was previously occupied by the object they were
previously pointing to).

Also use HeapVerifyBefore/AfterGC to get more visibility into the
problem.

best.
-- ramki

David Holmes wrote:
> Tony,
> 
> This is a GC question not a runtime question so I've cc'ed the GC list 
> and bcc'ed the runtime list.
> 
> David
> 
> Tony Guan said the following on 08/14/10 02:28:
>> Hi there,
>>
>> I wrote one collector of my own in hotspot, but now I have a problem.
>> In the new collector, I inserted another generation between the
>> defNewGeneration and TenuredGeneration. To collect this new
>> generation, I modified the defNewGeneration collector, which is gc by
>> copying survivors.
>>
>> After the collection on this generation, I am now having an Null
>> pointer exception in the java program. My guess is that the collector
>> failed to identify the live objects in the generation. Here I need
>> some one to tell me if there is any missed modification on the write
>> barrier or card marking.
>>
>> As far as I know, the write barrier works whenever a field write
>> happens, without regards to which generation is influenced. So this
>> means that the card of the written field will be marked. So my problem
>> should be in the closures that check if there are any live objects in
>> the generation. Am I right?
>>
>> Or other than the card scanning, is there anything special that I
>> should process?
>>
>> Thanks a lot!
>>
>>
>> Tony (Xiaohua Guan)


From guanxiaohua at gmail.com  Sun Aug 15 15:55:14 2010
From: guanxiaohua at gmail.com (Tony Guan)
Date: Sun, 15 Aug 2010 17:55:14 -0500
Subject: write barrier and card marking
In-Reply-To: <4C66DFA3.8060902@oracle.com>
References: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
	<4C65D3DE.5050803@oracle.com> <4C66DFA3.8060902@oracle.com>
Message-ID: <AANLkTikEFha6G4qwdnr5fYvBG7HC4tskeHzErjMEDeay@mail.gmail.com>

Hi David,

Thanks for the redirection!


Hi Ramki,

Thanks for the advice. I enabled verifying before and after GC. At the
very first full GC(also the first GC), the verification before the GC
passed, but the verification after GC failed.

Here is the error:
VerifyAfterGC:[Verifying threads permgen tenured generation
TransNewGeneration def new generation remset

Internal Error (/home/tony/software/OpenJDK/jdk7/hotspot/src/share/vm/memory/cardTableRS.cpp:319),
pid=16388, tid=1085323600
#  Error: guarantee(obj == __null || (HeapWord*)p < _boundary ||
(HeapWord*)obj >= _boundary,"pointer on clean card crosses boundary")


And the trace is:
V  [libjvm.so+0x8cc0b4];;  _ZN7VMError6reportEP12outputStream+0xb72
V  [libjvm.so+0x8cd1d4];;  _ZN7VMError14report_and_dieEv+0x5f6
V  [libjvm.so+0x403a85];;  _Z12report_fatalPKciS0_+0x6b
V  [libjvm.so+0x315636];;
_ZN22VerifyCleanCardClosure11do_oop_workIP7oopDescEEvPT_+0x82
V  [libjvm.so+0x31565b];;  _ZN22VerifyCleanCardClosure6do_oopEPP7oopDesc+0x1d
V  [libjvm.so+0x51ac5b];;
_ZN13instanceKlass21iterate_static_fieldsEP10OopClosure+0xbd
V  [libjvm.so+0x52e7ac];;
_ZN18instanceKlassKlass15oop_oop_iterateEP7oopDescP10OopClosure+0xa6
V  [libjvm.so+0x2a8590];;
_ZN5Klass17oop_oop_iterate_vEP7oopDescP10OopClosure+0x32
V  [libjvm.so+0x3108b9];;  _ZN7oopDesc11oop_iterateEP10OopClosure+0x37
V  [libjvm.so+0x314289];;
_ZN11CardTableRS12verify_spaceEP5SpaceP8HeapWord+0x34f
V  [libjvm.so+0x315507];;  _ZN20VerifyCTSpaceClosure8do_spaceEP5Space+0x29
V  [libjvm.so+0x4e5913];;
_ZN28OneContigSpaceCardGeneration13space_iterateEP12SpaceClosureb+0x35
V  [libjvm.so+0x39885d];;
_ZN20CompactingPermGenGen13space_iterateEP12SpaceClosureb+0x25
V  [libjvm.so+0x31450a];;  _ZN11CardTableRS6verifyEv+0x14e
V  [libjvm.so+0x4d5216];;  _ZN16GenCollectedHeap6verifyEbb+0x13a
V  [libjvm.so+0x89cfaf];;  _ZN8Universe6verifyEbb+0xfd
V  [libjvm.so+0x4d6f17];;
_ZN16GenCollectedHeap24do_type_based_collectionEbbmbib+0xb05
V  [libjvm.so+0x4d72a3];;  _ZN16GenCollectedHeap13do_collectionEbbmbi+0x93
V  [libjvm.so+0x4d7fda];;  _ZN16GenCollectedHeap18do_full_collectionEbi+0x78
V  [libjvm.so+0x8cde05];;  _ZN17VM_GenCollectFull4doitEv+0x5b
V  [libjvm.so+0x8e4f73];;  _ZN12VM_Operation8evaluateEv+0x5b
V  [libjvm.so+0x8e29a7];;  _ZN8VMThread18evaluate_operationEP12VM_Operation+0x33
V  [libjvm.so+0x8e2ef4];;  _ZN8VMThread4loopEv+0x4c0
V  [libjvm.so+0x8e33a6];;  _ZN8VMThread3runEv+0xf4
V  [libjvm.so+0x750f51];;  _Z10java_startP6Thread+0x16f

Now to answer your first question:
For minor collections, I collect the two generations below Tenured
separately.But a collection on old generation will just collect the
whole heap.(just like the original generational GC, the only
difference is like we have two young gens indexed as 0,1, and old
indexed as 2.)

This failures means that I failed to mark some card to be dirty when
there are inter-generational pointers in it.  And because the
pre-verification passed, the only reason for this failure is the full
collection. Now can you point to me that where could this error happen
during the full GC(marksweep)?

Thanks!


Tony (Xiaohua Guan)



On Sat, Aug 14, 2010 at 1:25 PM, Y. Srinivas Ramakrishna
<y.s.ramakrishna at oracle.com> wrote:
> Tony,
>
> Do you collect each generation independently, or
> does a generation at level n (with youngest == 0)
> collect all younger generations (i.e. level m < n)?
>
> The usual failure pattern for missed remembered set entries
> is that there is a "stale" pointer in a generation not collected
> but with a pointer into a generation that was collected.
> You can gainfully use the "mangling" feature in debug builds to
> identify such cases sometimes. I have not generally seen null pointer
> exceptions in such cases though (because that would mean that you are
> getting hold of a null reference, which requires more work, i.e. clearing,
> than forgetting to update an old reference to point to a new location),
> just "broken" references to stale objects (these usually end up pointing
> into the middle of unallocated space or into the middle of another object --
> i.e. a location that was previously occupied by the object they were
> previously pointing to).
>
> Also use HeapVerifyBefore/AfterGC to get more visibility into the
> problem.
>
> best.
> -- ramki
>
> David Holmes wrote:
>>
>> Tony,
>>
>> This is a GC question not a runtime question so I've cc'ed the GC list and
>> bcc'ed the runtime list.
>>
>> David
>>
>> Tony Guan said the following on 08/14/10 02:28:
>>>
>>> Hi there,
>>>
>>> I wrote one collector of my own in hotspot, but now I have a problem.
>>> In the new collector, I inserted another generation between the
>>> defNewGeneration and TenuredGeneration. To collect this new
>>> generation, I modified the defNewGeneration collector, which is gc by
>>> copying survivors.
>>>
>>> After the collection on this generation, I am now having an Null
>>> pointer exception in the java program. My guess is that the collector
>>> failed to identify the live objects in the generation. Here I need
>>> some one to tell me if there is any missed modification on the write
>>> barrier or card marking.
>>>
>>> As far as I know, the write barrier works whenever a field write
>>> happens, without regards to which generation is influenced. So this
>>> means that the card of the written field will be marked. So my problem
>>> should be in the closures that check if there are any live objects in
>>> the generation. Am I right?
>>>
>>> Or other than the card scanning, is there anything special that I
>>> should process?
>>>
>>> Thanks a lot!
>>>
>>>
>>> Tony (Xiaohua Guan)
>
>

From guanxiaohua at gmail.com  Sun Aug 15 16:59:20 2010
From: guanxiaohua at gmail.com (Tony Guan)
Date: Sun, 15 Aug 2010 18:59:20 -0500
Subject: write barrier and card marking
In-Reply-To: <AANLkTikEFha6G4qwdnr5fYvBG7HC4tskeHzErjMEDeay@mail.gmail.com>
References: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
	<4C65D3DE.5050803@oracle.com> <4C66DFA3.8060902@oracle.com>
	<AANLkTikEFha6G4qwdnr5fYvBG7HC4tskeHzErjMEDeay@mail.gmail.com>
Message-ID: <AANLkTin2DfLeeSAJKQPYAJ-jKy5jcvrrA=p4n__PM=ot@mail.gmail.com>

Hi Ramki,
I got the precise information about this failure:

[Verifying threads permgen tenured generation TransNewGeneration def
new generation remset:
 obj(0x7f107a5708f0)in gen(2) was pointed by p(0x7f107aae2450) in
gen(3), _boundary=0x7f107aad0000.

The _boundary is the perm starting address, so it's a pointer from
perm(gen 3) to tenured (gen 2). Since collection on old gen will also
collect perm gen, this should be fine, isn't it?

Tony (Xiaohua Guan)



On Sun, Aug 15, 2010 at 5:55 PM, Tony Guan <guanxiaohua at gmail.com> wrote:
> Hi David,
>
> Thanks for the redirection!
>
>
> Hi Ramki,
>
> Thanks for the advice. I enabled verifying before and after GC. At the
> very first full GC(also the first GC), the verification before the GC
> passed, but the verification after GC failed.
>
> Here is the error:
> VerifyAfterGC:[Verifying threads permgen tenured generation
> TransNewGeneration def new generation remset
>
> Internal Error (/home/tony/software/OpenJDK/jdk7/hotspot/src/share/vm/memory/cardTableRS.cpp:319),
> pid=16388, tid=1085323600
> # ?Error: guarantee(obj == __null || (HeapWord*)p < _boundary ||
> (HeapWord*)obj >= _boundary,"pointer on clean card crosses boundary")
>
>
> And the trace is:
> V ?[libjvm.so+0x8cc0b4];; ?_ZN7VMError6reportEP12outputStream+0xb72
> V ?[libjvm.so+0x8cd1d4];; ?_ZN7VMError14report_and_dieEv+0x5f6
> V ?[libjvm.so+0x403a85];; ?_Z12report_fatalPKciS0_+0x6b
> V ?[libjvm.so+0x315636];;
> _ZN22VerifyCleanCardClosure11do_oop_workIP7oopDescEEvPT_+0x82
> V ?[libjvm.so+0x31565b];; ?_ZN22VerifyCleanCardClosure6do_oopEPP7oopDesc+0x1d
> V ?[libjvm.so+0x51ac5b];;
> _ZN13instanceKlass21iterate_static_fieldsEP10OopClosure+0xbd
> V ?[libjvm.so+0x52e7ac];;
> _ZN18instanceKlassKlass15oop_oop_iterateEP7oopDescP10OopClosure+0xa6
> V ?[libjvm.so+0x2a8590];;
> _ZN5Klass17oop_oop_iterate_vEP7oopDescP10OopClosure+0x32
> V ?[libjvm.so+0x3108b9];; ?_ZN7oopDesc11oop_iterateEP10OopClosure+0x37
> V ?[libjvm.so+0x314289];;
> _ZN11CardTableRS12verify_spaceEP5SpaceP8HeapWord+0x34f
> V ?[libjvm.so+0x315507];; ?_ZN20VerifyCTSpaceClosure8do_spaceEP5Space+0x29
> V ?[libjvm.so+0x4e5913];;
> _ZN28OneContigSpaceCardGeneration13space_iterateEP12SpaceClosureb+0x35
> V ?[libjvm.so+0x39885d];;
> _ZN20CompactingPermGenGen13space_iterateEP12SpaceClosureb+0x25
> V ?[libjvm.so+0x31450a];; ?_ZN11CardTableRS6verifyEv+0x14e
> V ?[libjvm.so+0x4d5216];; ?_ZN16GenCollectedHeap6verifyEbb+0x13a
> V ?[libjvm.so+0x89cfaf];; ?_ZN8Universe6verifyEbb+0xfd
> V ?[libjvm.so+0x4d6f17];;
> _ZN16GenCollectedHeap24do_type_based_collectionEbbmbib+0xb05
> V ?[libjvm.so+0x4d72a3];; ?_ZN16GenCollectedHeap13do_collectionEbbmbi+0x93
> V ?[libjvm.so+0x4d7fda];; ?_ZN16GenCollectedHeap18do_full_collectionEbi+0x78
> V ?[libjvm.so+0x8cde05];; ?_ZN17VM_GenCollectFull4doitEv+0x5b
> V ?[libjvm.so+0x8e4f73];; ?_ZN12VM_Operation8evaluateEv+0x5b
> V ?[libjvm.so+0x8e29a7];; ?_ZN8VMThread18evaluate_operationEP12VM_Operation+0x33
> V ?[libjvm.so+0x8e2ef4];; ?_ZN8VMThread4loopEv+0x4c0
> V ?[libjvm.so+0x8e33a6];; ?_ZN8VMThread3runEv+0xf4
> V ?[libjvm.so+0x750f51];; ?_Z10java_startP6Thread+0x16f
>
> Now to answer your first question:
> For minor collections, I collect the two generations below Tenured
> separately.But a collection on old generation will just collect the
> whole heap.(just like the original generational GC, the only
> difference is like we have two young gens indexed as 0,1, and old
> indexed as 2.)
>
> This failures means that I failed to mark some card to be dirty when
> there are inter-generational pointers in it. ?And because the
> pre-verification passed, the only reason for this failure is the full
> collection. Now can you point to me that where could this error happen
> during the full GC(marksweep)?
>
> Thanks!
>
>
> Tony (Xiaohua Guan)
>
>
>
> On Sat, Aug 14, 2010 at 1:25 PM, Y. Srinivas Ramakrishna
> <y.s.ramakrishna at oracle.com> wrote:
>> Tony,
>>
>> Do you collect each generation independently, or
>> does a generation at level n (with youngest == 0)
>> collect all younger generations (i.e. level m < n)?
>>
>> The usual failure pattern for missed remembered set entries
>> is that there is a "stale" pointer in a generation not collected
>> but with a pointer into a generation that was collected.
>> You can gainfully use the "mangling" feature in debug builds to
>> identify such cases sometimes. I have not generally seen null pointer
>> exceptions in such cases though (because that would mean that you are
>> getting hold of a null reference, which requires more work, i.e. clearing,
>> than forgetting to update an old reference to point to a new location),
>> just "broken" references to stale objects (these usually end up pointing
>> into the middle of unallocated space or into the middle of another object --
>> i.e. a location that was previously occupied by the object they were
>> previously pointing to).
>>
>> Also use HeapVerifyBefore/AfterGC to get more visibility into the
>> problem.
>>
>> best.
>> -- ramki
>>
>> David Holmes wrote:
>>>
>>> Tony,
>>>
>>> This is a GC question not a runtime question so I've cc'ed the GC list and
>>> bcc'ed the runtime list.
>>>
>>> David
>>>
>>> Tony Guan said the following on 08/14/10 02:28:
>>>>
>>>> Hi there,
>>>>
>>>> I wrote one collector of my own in hotspot, but now I have a problem.
>>>> In the new collector, I inserted another generation between the
>>>> defNewGeneration and TenuredGeneration. To collect this new
>>>> generation, I modified the defNewGeneration collector, which is gc by
>>>> copying survivors.
>>>>
>>>> After the collection on this generation, I am now having an Null
>>>> pointer exception in the java program. My guess is that the collector
>>>> failed to identify the live objects in the generation. Here I need
>>>> some one to tell me if there is any missed modification on the write
>>>> barrier or card marking.
>>>>
>>>> As far as I know, the write barrier works whenever a field write
>>>> happens, without regards to which generation is influenced. So this
>>>> means that the card of the written field will be marked. So my problem
>>>> should be in the closures that check if there are any live objects in
>>>> the generation. Am I right?
>>>>
>>>> Or other than the card scanning, is there anything special that I
>>>> should process?
>>>>
>>>> Thanks a lot!
>>>>
>>>>
>>>> Tony (Xiaohua Guan)
>>
>>
>

From y.s.ramakrishna at oracle.com  Mon Aug 16 17:57:26 2010
From: y.s.ramakrishna at oracle.com (y.s.ramakrishna at oracle.com)
Date: Tue, 17 Aug 2010 00:57:26 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6948538: CMS: BOT walkers can fall into
	object allocation and initialization cracks
Message-ID: <20100817005727.E4CEC47225@hg.openjdk.java.net>

Changeset: be3f9c242c9d
Author:    ysr
Date:      2010-08-16 15:58 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/be3f9c242c9d

6948538: CMS: BOT walkers can fall into object allocation and initialization cracks
Summary: GC workers now recognize an intermediate transient state of blocks which are allocated but have not yet completed initialization. blk_start() calls do not attempt to determine the size of a block in the transient state, rather waiting for the block to become initialized so that it is safe to query its size. Audited and ensured the order of initialization of object fields (klass, free bit and size) to respect block state transition protocol. Also included some new assertion checking code enabled in debug mode.
Reviewed-by: chrisphi, johnc, poonam

! src/share/vm/gc_implementation/concurrentMarkSweep/compactibleFreeListSpace.cpp
! src/share/vm/gc_implementation/concurrentMarkSweep/compactibleFreeListSpace.hpp
! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp
! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.hpp
! src/share/vm/gc_implementation/concurrentMarkSweep/freeChunk.hpp
! src/share/vm/gc_implementation/concurrentMarkSweep/promotionInfo.cpp
! src/share/vm/gc_implementation/includeDB_gc_concurrentMarkSweep
! src/share/vm/includeDB_core
! src/share/vm/memory/blockOffsetTable.cpp
! src/share/vm/memory/blockOffsetTable.hpp
! src/share/vm/memory/blockOffsetTable.inline.hpp
! src/share/vm/runtime/globals.hpp


From y.s.ramakrishna at oracle.com  Mon Aug 16 18:17:00 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Mon, 16 Aug 2010 18:17:00 -0700
Subject: write barrier and card marking
In-Reply-To: <AANLkTikEFha6G4qwdnr5fYvBG7HC4tskeHzErjMEDeay@mail.gmail.com>
References: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
	<4C65D3DE.5050803@oracle.com> <4C66DFA3.8060902@oracle.com>
	<AANLkTikEFha6G4qwdnr5fYvBG7HC4tskeHzErjMEDeay@mail.gmail.com>
Message-ID: <4C69E30C.3060802@oracle.com>

Tony Guan wrote:

> This failures means that I failed to mark some card to be dirty when
> there are inter-generational pointers in it.  And because the
> pre-verification passed, the only reason for this failure is the full
> collection. Now can you point to me that where could this error happen
> during the full GC(marksweep)?

As far as i recall, following a full mark-sweep collection the
entire used section of a generation is marked dirty -- as long as there are
independently collectible generations, for example a young gen,
that has any surviving objects following that collection: this is a
conservative and simple, if slightly wasteful, strategy.
The next scavenge is therefore more expensive because of having to
deal with more dirty cards than required, but the redundant
card marks are sorted out and cleared in that collection. Perhaps you modified
the mark-sweep to be more discriminating during copying/updating (dirtying
only cards that have cross-generational references)?

i am sure you'll be able chase down your problem; all the best!
-- ramki

From y.s.ramakrishna at oracle.com  Mon Aug 16 18:22:05 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Mon, 16 Aug 2010 18:22:05 -0700
Subject: write barrier and card marking
In-Reply-To: <AANLkTin2DfLeeSAJKQPYAJ-jKy5jcvrrA=p4n__PM=ot@mail.gmail.com>
References: <AANLkTik8+gcEAmwsU9myC9MwR0B8cNyZ1M8NuNBZkR6y@mail.gmail.com>
	<4C65D3DE.5050803@oracle.com> <4C66DFA3.8060902@oracle.com>
	<AANLkTikEFha6G4qwdnr5fYvBG7HC4tskeHzErjMEDeay@mail.gmail.com>
	<AANLkTin2DfLeeSAJKQPYAJ-jKy5jcvrrA=p4n__PM=ot@mail.gmail.com>
Message-ID: <4C69E43D.7020807@oracle.com>

Tony Guan wrote:
> Hi Ramki,
> I got the precise information about this failure:
> 
> [Verifying threads permgen tenured generation TransNewGeneration def
> new generation remset:
>  obj(0x7f107a5708f0)in gen(2) was pointed by p(0x7f107aae2450) in
> gen(3), _boundary=0x7f107aad0000.
> 
> The _boundary is the perm starting address, so it's a pointer from
> perm(gen 3) to tenured (gen 2). Since collection on old gen will also
> collect perm gen, this should be fine, isn't it?

Right, if two "generations" are always collected together you would not
need to track pointer references between them. You would probably want
to relax the assert accordingly then, i am guessing. And then, hopefully,
the real problem will reveal itself (or your verification code can dump
all the violations and you can find the real one from the false one).

all the best.
-- ramki

From tony.printezis at oracle.com  Tue Aug 17 14:14:32 2010
From: tony.printezis at oracle.com (tony.printezis at oracle.com)
Date: Tue, 17 Aug 2010 21:14:32 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 3 new changesets
Message-ID: <20100817211437.773D047258@hg.openjdk.java.net>

Changeset: 688c3755d7af
Author:    tonyp
Date:      2010-08-17 14:40 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/688c3755d7af

6959014: G1: assert(minimum_desired_capacity <= maximum_desired_capacity) failed: sanity check
Summary: There are a few issues in the code that calculates whether to resize the heap and by how much: a) some calculations can overflow 32-bit size_t's, b) min_desired_capacity is not bounded by the max heap size, and c) the assrt that fires is in the wrong place. The fix also includes some tidying up of the related verbose code.
Reviewed-by: ysr, jmasa

! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp

Changeset: bb847e31b836
Author:    tonyp
Date:      2010-08-17 14:40 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/bb847e31b836

6974928: G1: sometimes humongous objects are allocated in young regions
Summary: as the title says, sometimes we are allocating humongous objects in young regions and we shouldn't.
Reviewed-by: ysr, johnc

! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.inline.hpp
! src/share/vm/gc_implementation/g1/heapRegion.cpp

Changeset: b63010841f78
Author:    tonyp
Date:      2010-08-17 14:40 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/b63010841f78

6975964: G1: print out a more descriptive message for evacuation failure when +PrintGCDetails is set
Summary: we're renaming "evacuation failure" to "to-space overflow". I'm also piggy-backing a small additional change which removes the "Mark closure took..." output.
Reviewed-by: ysr, johnc

! src/share/vm/gc_implementation/g1/concurrentMark.cpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp


From y.s.ramakrishna at oracle.com  Wed Aug 18 14:22:13 2010
From: y.s.ramakrishna at oracle.com (y.s.ramakrishna at oracle.com)
Date: Wed, 18 Aug 2010 21:22:13 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6977970: CMS:
	concurrentMarkSweepGeneration.cpp:7947 assert(addr <= _limit)
	failed: sweep invariant
Message-ID: <20100818212215.6135F47291@hg.openjdk.java.net>

Changeset: 5ed703250bff
Author:    ysr
Date:      2010-08-18 11:39 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5ed703250bff

6977970: CMS: concurrentMarkSweepGeneration.cpp:7947 assert(addr <= _limit) failed: sweep invariant
Summary: Allow for the possibility (when the heap is expanding) that the sweep might skip over and past, rather than necessarily step on, the sweep limit determined at the beginning of a concurrent marking cycle.
Reviewed-by: jmasa, tonyp

! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp


From john.cuthbertson at sun.com  Wed Aug 18 18:15:47 2010
From: john.cuthbertson at sun.com (john.cuthbertson at sun.com)
Date: Thu, 19 Aug 2010 01:15:47 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 2 new changesets
Message-ID: <20100819011552.3239B4729C@hg.openjdk.java.net>

Changeset: 413ad0331a0c
Author:    johnc
Date:      2010-08-18 10:59 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/413ad0331a0c

6977924: Changes for 6975078 produce build error with certain gcc versions
Summary: The changes introduced for 6975078 assign badHeapOopVal to the _allocation field in the ResourceObj class. In 32 bit linux builds with certain versions of gcc this assignment will be flagged as an error while compiling allocation.cpp. In 32 bit builds the constant value badHeapOopVal (which is cast to an intptr_t) is negative. The _allocation field is typed as an unsigned intptr_t and gcc catches this as an error.
Reviewed-by: jcoomes, ysr, phh

! src/share/vm/memory/allocation.cpp

Changeset: effb55808a18
Author:    johnc
Date:      2010-08-18 17:44 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/effb55808a18

Merge



From jacklusf at gmail.com  Thu Aug 19 12:44:24 2010
From: jacklusf at gmail.com (Xiaobin Lu)
Date: Thu, 19 Aug 2010 12:44:24 -0700
Subject: Threshold for PermGen cleanup
Message-ID: <AANLkTik02DRjuyRz2gz8arhT2wuTWu3oh3rvntr8HXEZ@mail.gmail.com>

Hi folks,

I have a question. Do we have a size threshold for perm gen before full GC
kicks in to clean it? Someone claimed to me that he saw perm gen usage goes
to 99.9% and he didn't see any full GC is triggered to collect it.

Thanks so much in advance!

-Xiaobin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100819/865a9a4f/attachment.html 

From y.s.ramakrishna at oracle.com  Thu Aug 19 13:05:30 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Thu, 19 Aug 2010 13:05:30 -0700
Subject: Threshold for PermGen cleanup
In-Reply-To: <AANLkTik02DRjuyRz2gz8arhT2wuTWu3oh3rvntr8HXEZ@mail.gmail.com>
References: <AANLkTik02DRjuyRz2gz8arhT2wuTWu3oh3rvntr8HXEZ@mail.gmail.com>
Message-ID: <4C6D8E8A.4080509@oracle.com>

Hi Xiaobin --

Look at the method PermGen::mem_allocate_in_gen() to see what we
do. We allow expansion up to a (tunable) delta without a collection,
when we reach the delta, we do a collection (and possibly resize),
and the cycle repeats. There is no threshold occupancy at which
a collection is done -- the decision to expand or collect is
made when an allocation fails. It is possible that "he" was
running with MaxPermSize and PermSize set to the same value,
and yes in that case no full collection will be triggered until
the perm gen (or the old gen) gets full. In practice, since in
typical deployments the old gen allocation rates are usually much higher,
the old gen typically will get full before perm does and this
will precipitate a whole heap collection during
which the perm gen is (always, modulo CMS) also collected.

Or were you talking about the concurrent CMS collector where the
policy was designed to be different (but has a performance
bug/regression that we know of but, alas, haven't gotten around to
fixing yet).

-- ramki


Xiaobin Lu wrote:
> Hi folks,
> 
> I have a question. Do we have a size threshold for perm gen before full GC
> kicks in to clean it? Someone claimed to me that he saw perm gen usage goes
> to 99.9% and he didn't see any full GC is triggered to collect it.
> 
> Thanks so much in advance!
> 
> -Xiaobin
> 


From John.Coomes at oracle.com  Thu Aug 19 16:54:20 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Thu, 19 Aug 2010 16:54:20 -0700
Subject: review request (XS): 6978300 G1 debug builds crash if ParGCThreads==0
Message-ID: <19565.50220.147050.857692@oracle.com>

I'd appreciate reviews of this small change:

http://cr.openjdk.java.net/~jcoomes/6978300-taskqueue-g1-gcthreads/

Debug builds of G1 crash when ParallelGCThreads==0 because of a null
pointer deref.  It was introduced by my changes for 6966222.

-John


From y.s.ramakrishna at oracle.com  Thu Aug 19 17:46:43 2010
From: y.s.ramakrishna at oracle.com (y.s.ramakrishna at oracle.com)
Date: Fri, 20 Aug 2010 00:46:43 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6978533: CMS: Elide BOT update asserts
	until 6977974 is fixed correctly
Message-ID: <20100820004654.C05C4472D5@hg.openjdk.java.net>

Changeset: 52f2bc645da5
Author:    ysr
Date:      2010-08-19 12:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/52f2bc645da5

6978533: CMS: Elide BOT  update asserts until 6977974 is fixed correctly
Reviewed-by: jcoomes, jmasa, tonyp

! src/share/vm/memory/blockOffsetTable.hpp


From tony.printezis at oracle.com  Mon Aug 23 09:00:43 2010
From: tony.printezis at oracle.com (tony.printezis at oracle.com)
Date: Mon, 23 Aug 2010 16:00:43 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 29 new changesets
Message-ID: <20100823160136.7E54F473B8@hg.openjdk.java.net>

Changeset: cb4250ef73b2
Author:    mikejwre
Date:      2010-07-23 16:42 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/cb4250ef73b2

Added tag jdk7-b102 for changeset c5cadf1a0771

! .hgtags

Changeset: efd4401fab1d
Author:    cl
Date:      2010-07-29 13:33 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/efd4401fab1d

Added tag jdk7-b103 for changeset cb4250ef73b2

! .hgtags

Changeset: cc3fdfeb54b0
Author:    trims
Date:      2010-07-29 23:14 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/cc3fdfeb54b0

Merge


Changeset: fd2645290e89
Author:    trims
Date:      2010-07-30 06:56 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/fd2645290e89

6973381: Bump the HS19 build number to 05
Summary: Update the HS19 build number to 05
Reviewed-by: jcoomes

! make/hotspot_version

Changeset: 28abe3f6a5f6
Author:    trims
Date:      2010-08-03 19:01 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/28abe3f6a5f6

Merge


Changeset: b4acf10eb134
Author:    trims
Date:      2010-08-05 02:48 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/b4acf10eb134

Added tag hs19-b04 for changeset e55900b5c1b8

! .hgtags

Changeset: 6709c14587c2
Author:    cl
Date:      2010-08-06 12:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6709c14587c2

Added tag jdk7-b104 for changeset b4acf10eb134

! .hgtags

Changeset: 3dc64719cf18
Author:    cl
Date:      2010-08-13 11:38 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/3dc64719cf18

Added tag jdk7-b105 for changeset 6709c14587c2

! .hgtags

Changeset: 688a538aa654
Author:    trims
Date:      2010-08-13 10:55 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/688a538aa654

Merge


Changeset: 5f3c8db59d83
Author:    trims
Date:      2010-08-13 10:56 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5f3c8db59d83

6977051: Bump the HS19 build number to 06
Summary: Update the HS19 build number to 06
Reviewed-by: jcoomes

! make/hotspot_version

Changeset: 1b81ca701fa5
Author:    trims
Date:      2010-08-17 09:43 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1b81ca701fa5

Merge


Changeset: f121b2772674
Author:    trims
Date:      2010-08-18 16:11 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f121b2772674

Merge

- src/share/vm/gc_implementation/parallelScavenge/prefetchQueue.hpp

Changeset: 495caa35b1b5
Author:    asaha
Date:      2010-08-17 22:52 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/495caa35b1b5

6977952: Test: Sync missing tests from hs16.3 to hs17.x
Reviewed-by: wrockett

+ test/compiler/6894807/IsInstanceTest.java
+ test/compiler/6894807/Test6894807.sh
+ test/runtime/6626217/IFace.java
+ test/runtime/6626217/Loader2.java
+ test/runtime/6626217/Test6626217.sh
+ test/runtime/6626217/You_Have_Been_P0wned.java
+ test/runtime/6626217/bug_21227.java
+ test/runtime/6626217/from_loader2.java
+ test/runtime/6626217/many_loader1.java.foo
+ test/runtime/6626217/many_loader2.java.foo

Changeset: 1b0104ab1e5e
Author:    tonyp
Date:      2010-08-19 14:08 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1b0104ab1e5e

Merge


Changeset: 30266066c77c
Author:    cl
Date:      2010-08-19 15:13 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/30266066c77c

Added tag jdk7-b106 for changeset 1b81ca701fa5

! .hgtags

Changeset: 295c3ae4ab5b
Author:    trims
Date:      2010-08-19 18:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/295c3ae4ab5b

Added tag hs19-b05 for changeset cc3fdfeb54b0

! .hgtags

Changeset: bf496cbe9b74
Author:    trims
Date:      2010-08-19 18:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/bf496cbe9b74

Added tag hs19-b06 for changeset 688a538aa654

! .hgtags

Changeset: 0e509ddd9962
Author:    trims
Date:      2010-08-20 03:47 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0e509ddd9962

6978726: Bump the HS19 build number to 07
Summary: Update the HS19 build number to 07
Reviewed-by: jcoomes

! make/hotspot_version

Changeset: 09cdb1e1c77b
Author:    trims
Date:      2010-08-20 04:08 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/09cdb1e1c77b

Merge

- src/share/vm/gc_implementation/parallelScavenge/prefetchQueue.hpp

Changeset: 71faaa8e3ccc
Author:    never
Date:      2010-08-12 16:38 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/71faaa8e3ccc

6974176: ShouldNotReachHere, instanceKlass.cpp:1426
Reviewed-by: kvn, twisti

! src/share/vm/code/nmethod.cpp
! src/share/vm/code/nmethod.hpp

Changeset: da877bdc9000
Author:    never
Date:      2010-08-12 23:34 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/da877bdc9000

6975006: assert(check.is_deoptimized_frame()) failed: missed deopt
Reviewed-by: kvn, twisti

! src/share/vm/runtime/frame.cpp
! src/share/vm/runtime/frame.hpp
! src/share/vm/runtime/safepoint.cpp
! src/share/vm/runtime/safepoint.hpp
! src/share/vm/runtime/thread.cpp

Changeset: a62d332029cf
Author:    never
Date:      2010-08-13 15:14 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a62d332029cf

6976372: #  assert(_owner == Thread::current()) failed: invariant
Reviewed-by: kvn, twisti

! src/share/vm/compiler/compileBroker.cpp
! src/share/vm/runtime/sharedRuntime.cpp

Changeset: 13b87063b4d8
Author:    twisti
Date:      2010-08-18 01:22 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/13b87063b4d8

6977640: Zero and Shark fixes
Summary: A number of fixes for Zero and Shark.
Reviewed-by: twisti
Contributed-by: Gary Benson <gbenson at redhat.com>

! src/cpu/zero/vm/bytecodeInterpreter_zero.inline.hpp
! src/cpu/zero/vm/javaFrameAnchor_zero.hpp
! src/os_cpu/linux_zero/vm/os_linux_zero.cpp
! src/os_cpu/linux_zero/vm/thread_linux_zero.cpp
! src/share/vm/interpreter/bytecodeInterpreter.cpp

Changeset: f55c4f82ab9d
Author:    never
Date:      2010-08-19 14:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f55c4f82ab9d

6978249: spill between cpu and fpu registers when those moves are fast
Reviewed-by: kvn

! src/cpu/sparc/vm/vm_version_sparc.cpp
! src/cpu/x86/vm/vm_version_x86.cpp
! src/cpu/x86/vm/x86_32.ad
! src/cpu/x86/vm/x86_64.ad
! src/share/vm/opto/c2_globals.hpp
! src/share/vm/opto/coalesce.cpp
! src/share/vm/opto/matcher.cpp
! src/share/vm/opto/reg_split.cpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/init.cpp

Changeset: ee5cc9e78493
Author:    never
Date:      2010-08-20 09:55 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/ee5cc9e78493

Merge

! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/thread.cpp

Changeset: 66b9f90a9211
Author:    tonyp
Date:      2010-08-20 13:17 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/66b9f90a9211

Merge


Changeset: 26faca352942
Author:    tonyp
Date:      2010-08-20 12:01 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/26faca352942

Merge


Changeset: 571f6b35140b
Author:    trims
Date:      2010-08-20 12:57 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/571f6b35140b

6978889: Remove premature change of build number to Hotspot 19 Build 07
Summary: Change the build number back to 06
Reviewed-by: jcoomes

! make/hotspot_version

Changeset: b0b9d64ed9bc
Author:    trims
Date:      2010-08-20 14:24 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/b0b9d64ed9bc

6978915: Remove Mercurial tags for Hotspot 19 Build 06
Summary: Delete the hs19-b06 Hg tag, as it was put on incorrectly
Reviewed-by: jcoomes

! .hgtags


From John.Coomes at oracle.com  Mon Aug 23 17:08:46 2010
From: John.Coomes at oracle.com (John Coomes)
Date: Mon, 23 Aug 2010 17:08:46 -0700
Subject: review request (XS): 6978300 G1 debug builds crash if
	ParGCThreads==0
In-Reply-To: <19565.50220.147050.857692@oracle.com>
References: <19565.50220.147050.857692@oracle.com>
Message-ID: <19571.3470.745203.526862@oracle.com>

John Coomes (John.Coomes at oracle.com) wrote:
> I'd appreciate reviews of this small change:
> 
> http://cr.openjdk.java.net/~jcoomes/6978300-taskqueue-g1-gcthreads/
> 
> Debug builds of G1 crash when ParallelGCThreads==0 because of a null
> pointer deref.  It was introduced by my changes for 6966222.

As suggested by two reviewers, I changed the code to print out stats
for the single available thread when ParallelGCThreads==0, instead of
printing nothing.  The diffs are now down to 2 lines :-).

Updated webrev is at the url above.

-John


From y.s.ramakrishna at oracle.com  Tue Aug 24 09:08:13 2010
From: y.s.ramakrishna at oracle.com (y.s.ramakrishna at oracle.com)
Date: Tue, 24 Aug 2010 16:08:13 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6910183: CMS: assert(_index < capacity(),
	"_index out of bounds")
Message-ID: <20100824160824.69022473F5@hg.openjdk.java.net>

Changeset: bba76f745fe6
Author:    ysr
Date:      2010-08-23 17:51 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/bba76f745fe6

6910183: CMS: assert(_index < capacity(),"_index out of bounds")
Summary: Weakened a too-strong, off-by-one assert; added code to keep track of and report any overflows at appropriate level of verbosity.
Reviewed-by: jcoomes, tonyp

! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.hpp


From tony.printezis at oracle.com  Wed Aug 25 09:24:21 2010
From: tony.printezis at oracle.com (tony.printezis at oracle.com)
Date: Wed, 25 Aug 2010 16:24:21 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6941275: G1: The MemoryPools are
	incorrectly supported for G1
Message-ID: <20100825162425.63CAC4742E@hg.openjdk.java.net>

Changeset: e967bad2a9ab
Author:    tonyp
Date:      2010-08-25 08:44 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/e967bad2a9ab

6941275: G1: The MemoryPools are incorrectly supported for G1
Summary: The way we were caluclating the max value meant that it might fluctuate during the run and this broke some assumptions inside the MBeans framework. This change sets the max value of each pool to -1, which means undefined according to the spec.
Reviewed-by: mchung, johnc

! src/share/vm/services/g1MemoryPool.cpp
! src/share/vm/services/g1MemoryPool.hpp


From john.coomes at oracle.com  Wed Aug 25 16:44:01 2010
From: john.coomes at oracle.com (john.coomes at oracle.com)
Date: Wed, 25 Aug 2010 23:44:01 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6978300: G1: debug builds crash if
	ParallelGCThreads==0
Message-ID: <20100825234403.624FA47443@hg.openjdk.java.net>

Changeset: 8e5955ddf8e4
Author:    jcoomes
Date:      2010-08-25 14:39 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/8e5955ddf8e4

6978300: G1: debug builds crash if ParallelGCThreads==0
Reviewed-by: tonyp, ysr

! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp


From kevin.walls at sun.com  Fri Aug 27 09:09:14 2010
From: kevin.walls at sun.com (kevin.walls at sun.com)
Date: Fri, 27 Aug 2010 16:09:14 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 6980392: TEST_BUG:
	gc/6581734/Test6581734.java has typo
Message-ID: <20100827160918.42F8A474B3@hg.openjdk.java.net>

Changeset: 21c29458b334
Author:    kevinw
Date:      2010-08-27 16:57 +0100
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/21c29458b334

6980392: TEST_BUG: gc/6581734/Test6581734.java has typo
Summary: simple correction in testcase
Reviewed-by: mchung

! test/gc/6581734/Test6581734.java


From tony.printezis at oracle.com  Fri Aug 27 17:04:02 2010
From: tony.printezis at oracle.com (tony.printezis at oracle.com)
Date: Sat, 28 Aug 2010 00:04:02 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 2 new changesets
Message-ID: <20100828000406.9693B474CC@hg.openjdk.java.net>

Changeset: 1c63587d925b
Author:    tonyp
Date:      2010-08-27 13:34 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1c63587d925b

6980206: G1: assert(has_undefined_max_size, "Undefined max size");
Summary: An assert in the management.cpp is too strong and assumes the max size is always defined on memory pools, even when we don't need to use it.
Reviewed-by: mchung, johnc

! src/share/vm/services/management.cpp

Changeset: af586a7893cf
Author:    tonyp
Date:      2010-08-27 10:44 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/af586a7893cf

Merge



From shane.cox at gmail.com  Mon Aug 30 08:20:59 2010
From: shane.cox at gmail.com (Shane Cox)
Date: Mon, 30 Aug 2010 11:20:59 -0400
Subject: SIGSEGV during Minor GC
Message-ID: <AANLkTik_yHfV2oFm6FmzPHoRW6G+3mdkueZ3O0ayof9j@mail.gmail.com>

We recently had a JVM crash during Minor GC.  The hotspot error file and gc
log can be found in attached zip file.  GC settings are:
-Xms350m -Xmx1024m -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=55 -XX:+DisableExplicitGC -XX:+PrintGC
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:RefDiscoveryPolicy=1

We're running build:
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)


Any ideas what caused the crash?  Some speculate bug 6896647, but it doesn't
seem conclusive.  Any insights would be appreciated.

Thanks,
Shane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100830/91b62d51/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logs.zip
Type: application/zip
Size: 112478 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100830/91b62d51/attachment-0001.zip 

From y.s.ramakrishna at oracle.com  Mon Aug 30 08:54:04 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Mon, 30 Aug 2010 08:54:04 -0700
Subject: SIGSEGV during Minor GC
In-Reply-To: <AANLkTik_yHfV2oFm6FmzPHoRW6G+3mdkueZ3O0ayof9j@mail.gmail.com>
References: <AANLkTik_yHfV2oFm6FmzPHoRW6G+3mdkueZ3O0ayof9j@mail.gmail.com>
Message-ID: <4C7BD41C.5090909@oracle.com>

Hi Shane --

Shane Cox wrote:
> We recently had a JVM crash during Minor GC.  The hotspot error file and gc
> log can be found in attached zip file.  GC settings are:
> -Xms350m -Xmx1024m -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=55 -XX:+DisableExplicitGC -XX:+PrintGC
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:RefDiscoveryPolicy=1
> 
> We're running build:
> java version "1.6.0_14"
> Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
> Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
> 
> 
> Any ideas what caused the crash?  Some speculate bug 6896647, but it doesn't
> seem conclusive.  Any insights would be appreciated.

I don't think this is 6896647 (or at least it doesn't the classic symptoms of
that bug).

I'd recommend reporting this in through the official support channels so
that they can help diagnose and resolve the issue for you. Of course 6u14 is
rather old by now, so you might consider a newer version as well.

I could not immediately see it being related to any specific recent bug that we
fixed in CMS, but there are a few fixes that have gone in recently for bugs which could,
under appropriate circumstances, corrupt the free list (which this crash resembles).

-- ramki
> 
> Thanks,
> Shane


From y.s.ramakrishna at oracle.com  Mon Aug 30 11:28:40 2010
From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna)
Date: Mon, 30 Aug 2010 11:28:40 -0700
Subject: negative pause times
In-Reply-To: <4C6CFECC.7050002@AngelikaLanger.com>
References: <4C6CFECC.7050002@AngelikaLanger.com>
Message-ID: <4C7BF858.5000105@oracle.com>

Angelika Langer wrote:
> Hi!
> 
> Does anybody know off hand why the G1 log file shows negative pause times?
> 
> 
> 222.121: [GC pause (young) 561M->528M(1005M), 0.0586070 secs]
> 222.915: [GC pause (young) 576M->540M(1005M), 0.0447820 secs]
> 223.763: [GC pause (young) 602M->562M(1005M), -0.7354870 secs]  <=======
> 225.572: [GC pause (young) 772M->632M(1005M), 0.0719350 secs]
> 

Is this on Linux? Are you running NTP or otherwise causing TOD to
be adjusted? (you might want to turn off NTP if the negative times
bother you.) Are you running on VMware? (try a non-virtualized
run to see if the problem reproduces.) The negative pause times
we have seen in the past have usually been related to one of the
other of the above.

Please file a formal bug using your support credentials should
you believe this is an issue that needs delving deeper, especially
if you have a usable test case.

thanks.
-- ramki


> 
> Thanks
> Angelika Langer
> 
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From tony.printezis at oracle.com  Tue Aug 31 02:34:16 2010
From: tony.printezis at oracle.com (tony.printezis at oracle.com)
Date: Tue, 31 Aug 2010 09:34:16 +0000
Subject: hg: jdk7/hotspot-gc/hotspot: 7 new changesets
Message-ID: <20100831093428.60E1C4757A@hg.openjdk.java.net>

Changeset: f8c5d1bdaad4
Author:    ptisnovs
Date:      2010-08-19 14:23 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f8c5d1bdaad4

6885308: The incorrect -XX:StackRedPages, -XX:StackShadowPages, -XX:StackYellowPages could cause VM crash
Summary: Test minimal stack sizes given (also fixed linux compilation error)
Reviewed-by: never, phh, coleenp

! src/share/vm/memory/allocation.cpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/arguments.hpp

Changeset: ebfb7c68865e
Author:    dcubed
Date:      2010-08-23 08:44 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/ebfb7c68865e

Merge

! src/share/vm/memory/allocation.cpp
! src/share/vm/runtime/arguments.cpp

Changeset: 4b29a725c43c
Author:    jrose
Date:      2010-08-20 23:40 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/4b29a725c43c

6912064: type profiles need to be exploited more for dynamic language support
Reviewed-by: kvn

! src/share/vm/includeDB_compiler2
! src/share/vm/opto/graphKit.cpp
! src/share/vm/opto/graphKit.hpp
! src/share/vm/opto/library_call.cpp
! src/share/vm/opto/parse.hpp
! src/share/vm/opto/parse2.cpp
! src/share/vm/opto/parseHelper.cpp
! src/share/vm/runtime/globals.hpp

Changeset: 53dbe853fb3a
Author:    kvn
Date:      2010-08-23 09:09 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/53dbe853fb3a

6896381: CTW fails share/vm/ci/bcEscapeAnalyzer.cpp:99, assert(_stack_height < _max_stack,"stack overflow")
Summary: Check constant Tag type instead of calling get_constant().
Reviewed-by: never

! src/share/vm/ci/bcEscapeAnalyzer.cpp

Changeset: 3e8fbc61cee8
Author:    twisti
Date:      2010-08-25 05:27 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/3e8fbc61cee8

6978355: renaming for 6961697
Summary: This is the renaming part of 6961697 to keep the actual changes small for review.
Reviewed-by: kvn, never

! agent/src/share/classes/sun/jvm/hotspot/CommandProcessor.java
! agent/src/share/classes/sun/jvm/hotspot/c1/Runtime1.java
! agent/src/share/classes/sun/jvm/hotspot/code/CodeBlob.java
! agent/src/share/classes/sun/jvm/hotspot/code/NMethod.java
! agent/src/share/classes/sun/jvm/hotspot/code/PCDesc.java
! agent/src/share/classes/sun/jvm/hotspot/ui/FindInCodeCachePanel.java
! agent/src/share/classes/sun/jvm/hotspot/ui/classbrowser/HTMLGenerator.java
! agent/src/share/classes/sun/jvm/hotspot/utilities/PointerFinder.java
! agent/src/share/classes/sun/jvm/hotspot/utilities/PointerLocation.java
! src/cpu/sparc/vm/assembler_sparc.cpp
! src/cpu/sparc/vm/codeBuffer_sparc.hpp
! src/cpu/sparc/vm/frame_sparc.cpp
! src/cpu/sparc/vm/jniFastGetField_sparc.cpp
! src/cpu/sparc/vm/nativeInst_sparc.cpp
! src/cpu/sparc/vm/sparc.ad
! src/cpu/x86/vm/frame_x86.cpp
! src/cpu/x86/vm/frame_x86.inline.hpp
! src/cpu/x86/vm/jniFastGetField_x86_32.cpp
! src/cpu/x86/vm/jniFastGetField_x86_64.cpp
! src/cpu/x86/vm/vm_version_x86.cpp
! src/cpu/x86/vm/x86_32.ad
! src/cpu/x86/vm/x86_64.ad
! src/os/solaris/dtrace/generateJvmOffsets.cpp
! src/os/solaris/dtrace/libjvm_db.c
! src/os_cpu/windows_x86/vm/os_windows_x86.cpp
! src/os_cpu/windows_x86/vm/windows_x86_32.ad
! src/os_cpu/windows_x86/vm/windows_x86_64.ad
! src/share/vm/adlc/output_c.cpp
! src/share/vm/asm/codeBuffer.cpp
! src/share/vm/asm/codeBuffer.hpp
! src/share/vm/c1/c1_Compilation.cpp
! src/share/vm/c1/c1_Runtime1.cpp
! src/share/vm/c1/c1_Runtime1.hpp
! src/share/vm/ci/ciMethod.cpp
! src/share/vm/code/codeBlob.cpp
! src/share/vm/code/codeBlob.hpp
! src/share/vm/code/codeCache.cpp
! src/share/vm/code/exceptionHandlerTable.cpp
! src/share/vm/code/nmethod.cpp
! src/share/vm/code/nmethod.hpp
! src/share/vm/code/pcDesc.cpp
! src/share/vm/code/relocInfo.cpp
! src/share/vm/code/scopeDesc.cpp
! src/share/vm/code/stubs.cpp
! src/share/vm/code/vtableStubs.cpp
! src/share/vm/compiler/compileBroker.cpp
! src/share/vm/compiler/disassembler.cpp
! src/share/vm/interpreter/interpreter.hpp
! src/share/vm/interpreter/interpreterRuntime.cpp
! src/share/vm/opto/compile.cpp
! src/share/vm/opto/graphKit.cpp
! src/share/vm/opto/lcm.cpp
! src/share/vm/opto/library_call.cpp
! src/share/vm/opto/output.cpp
! src/share/vm/opto/stringopts.cpp
! src/share/vm/prims/jvmtiCodeBlobEvents.cpp
! src/share/vm/prims/jvmtiExport.cpp
! src/share/vm/prims/methodHandles.cpp
! src/share/vm/runtime/compilationPolicy.cpp
! src/share/vm/runtime/frame.cpp
! src/share/vm/runtime/icache.cpp
! src/share/vm/runtime/rframe.cpp
! src/share/vm/runtime/sharedRuntime.cpp
! src/share/vm/runtime/sharedRuntime.hpp
! src/share/vm/runtime/stubRoutines.cpp
! src/share/vm/runtime/vmStructs.cpp

Changeset: b4099f5786da
Author:    never
Date:      2010-08-25 10:31 -0700
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/b4099f5786da

Merge

! src/share/vm/runtime/globals.hpp

Changeset: 75107ee8712f
Author:    tonyp
Date:      2010-08-30 13:00 -0400
URL:       http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/75107ee8712f

Merge



From jon.masamitsu at oracle.com  Tue Aug 31 11:28:52 2010
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Tue, 31 Aug 2010 11:28:52 -0700
Subject: Long GC pause time
In-Reply-To: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>
References: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>
Message-ID: <4C7D49E4.6060908@oracle.com>

Try -XX+ParallelRefProcEnabled

Your spending much of the pause time doing Reference processing

89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 
0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
[Times: user=1.68 sys=0.01, real=1.41 secs]

(see the "[weak refs processing, 1.3860230 secs]") so doing it in 
parallel might
help.  ParallelRefProcEnabled isn't always a win so it's not "on" by 
default.

You might also try -XX:+CMSScavengeBeforeRemark.  That's a just a guess.
Sometimes doing a minor collection before the remark helps.

On 08/31/10 08:22, noam katzowicz wrote:
> Hello.
>  
> We are running JBoss AS over RH5, using java 6u21, with the following 
> GC params:
>  
> -Xms6750m -Xmx6750m
> -XX:+UseConcMarkSweepGC
> -XX:SurvivorRatio=128
> -XX:MaxTenuringThreshold=0
> -XX:+UseCompressedOops
> -XX:CMSInitiatingOccupancyFraction=60
> -Dsun.rmi.dgc.client.gcInterval=3600000
> -Dsun.rmi.dgc.server.gcInterval=3600000
> -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
>  
> We are experiencing relatively long GC pause time of over 1 second.
> Our application allocates hundreds of MBs per second. We?ve tried 
> different generation sizes but the above was the best we came up with. 
> Your help is appreciated.
>  
>
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 
> 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
> 89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 
> sys=2.30, real=1.65 secs]
> 89.487: [CMS-concurrent-abortable-preclean-start]
> 89.804: [GC 89.804: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 
> 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs]
> 89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: 
> user=2.42 sys=0.54, real=0.38 secs]
> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) 
> , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
> [Times: user=1.68 sys=0.01, real=1.41 secs]
> 91.284: [CMS-concurrent-sweep-start]
> 91.590: [GC 91.591: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 
> 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]
> 91.860: [GC 91.860: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 
> 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
> 92.786: [GC 92.787: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 
> 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs]
> 93.106: [GC 93.107: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 
> 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]
> 93.371: [GC 93.371: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 
> 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
> 94.637: [GC 94.638: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 
> 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs]
> 94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 
> sys=3.23, real=3.58 secs]
> 94.863: [CMS-concurrent-reset-start]
> 94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 
> sys=0.01, real=0.02 secs]
> 95.981: [GC 95.981: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 
> 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs]
> 100.179: [GC 100.180: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 
> 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs]
> 100.909: [GC 100.909: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 
> 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs]
> 101.613: [GC 101.613: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 
> 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
> 102.072: [GC 102.073: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 
> 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs]
> 102.756: [GC 102.756: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 
> 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
> 103.452: [GC 103.453: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 
> 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
> 104.462: [GC 104.462: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 
> 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs]
> 106.274: [GC 106.274: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 
> 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs]
> 108.154: [GC 108.155: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 
> 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs]
> 108.882: [GC 108.883: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 
> 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs]
> 109.482: [GC 109.482: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 
> 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs]
> 109.948: [GC 109.949: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 
> 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs]
> 110.035: [GC 110.035: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 
> 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
> 110.484: [GC 110.485: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 
> 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs]
> 110.910: [GC 110.910: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 
> 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs]
> 111.276: [GC 111.277: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 
> 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs]
> 111.658: [GC 111.659: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 
> 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs]
> 112.068: [GC 112.068: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 
> 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs]
> 112.497: [GC 112.498: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 
> 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs]
> 112.911: [GC 112.911: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 
> 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs]
> 113.362: [GC 113.363: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 
> 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs]
> 113.829: [GC 113.829: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 
> 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs]
> 114.311: [GC 114.311: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 
> 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
> 114.841: [GC 114.842: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 
> 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
> 115.356: [GC 115.357: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 
> 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
> 115.829: [GC 115.830: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 
> 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs]
> 116.271: [GC 116.271: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 
> 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs]
> 116.716: [GC 116.717: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 
> 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]
> 117.145: [GC 117.146: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 
> 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs]
> 117.543: [GC 117.545: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 
> 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs]
> 117.956: [GC 117.956: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 
> 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs]
> 118.385: [GC 118.386: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 
> 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
> 118.826: [GC 118.827: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 
> 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs]
> 119.233: [GC 119.234: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 
> 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs]
> 119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 
> 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, 
> real=0.01 secs]
> 119.311: [CMS-concurrent-mark-start]
> 119.699: [GC 119.699: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 
> 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs]
> 120.081: [GC 120.082: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 
> 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
> 120.538: [GC 120.539: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 
> 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs]
> 120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 
> sys=1.73, real=1.31 secs]
> 120.621: [CMS-concurrent-preclean-start]
> 120.942: [GC 120.943: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 
> 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs]
> 121.337: [GC 121.338: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 
> 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs]
> 121.766: [GC 121.767: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 
> 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
> 122.174: [GC 122.175: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 
> 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs]
> 122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: 
> user=10.25 sys=2.21, real=1.63 secs]
> 122.257: [CMS-concurrent-abortable-preclean-start]
> 122.631: [GC 122.631: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 
> 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs]
> 122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: 
> user=2.50 sys=0.52, real=0.44 secs]
> 122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan 
> (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 
> secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 
> secs] [Times: user=1.42 sys=0.01, real=1.14 secs]
> 123.834: [CMS-concurrent-sweep-start]
> 124.060: [GC 124.061: [ParNew
> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
> : 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 
> 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100831/d6e7cd34/attachment-0001.html 
-------------- next part --------------
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From adamh at basis.com  Tue Aug 31 12:36:53 2010
From: adamh at basis.com (Adam Hawthorne)
Date: Tue, 31 Aug 2010 15:36:53 -0400
Subject: iCMS question
Message-ID: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>

Hi all,

I'm looking at an i-cms log, and I don't understand why I'm seeing what I'm
seeing.  Any help is appreciated.  At about 38718.035 in the log below,
icms_dc begins to report 100, and just after that I see several back-to-back
concurrent mode failures.  During that time, the ParNew collections are
reporting that they're doing no work.  This happens on and off for about 20
minutes in the log.  Obviously the collections are taking seconds, but I
don't see a Full GC every time.  What is it doing when it's not doing a full
GC?  Is the continual CMS collection preventing the young generation from
being collected (or perhaps collecting it simultaneously)?  When
icms_dc=100, does that imply the application is stopped?

There was some uninformed command line twiddling going on here, so the GC
parameters were:
-Xmx1152m
-Xms256m
-XX:NewRatio=4
-XX:MaxPermSize=128m
-XX:+UseParNewGC
-XX:-CMSParallelRemarkEnabled
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=1
-XX:+CMSClassUnloadingEnabled
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-XX:CMSInitiatingOccupancyFraction=50
-XX:ParallelGCThreads=1
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=1024
-XX:SoftRefLRUPolicyMSPerMB=1000
-XX:+DisableExplicitGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps

I had them remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0,
and -XX:SurvivorRatio=1024 , thinking that since the duty cycle was
approaching 100%, and because the ParNew pause times were so low, we could
probably afford to spend more time doing young gc's instead of promoting.
 I'm thinking I should probably tell them to increase their -Xmx and
decrease their NewRatio as well.  Is there anything else?  I haven't done
much tuning of i-cms in general, but the standard i-cms safety options don't
seem like they would help here.

Here's the log excerpt:

2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew:
235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K) icms_dc=41
, 0.0189305 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark:
923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew:
235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K) icms_dc=41
, 0.0197785 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew:
235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K) icms_dc=41
, 0.0201448 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew:
235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K) icms_dc=56
, 0.0551800 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark: 1.903/4.332
secs] [Times: user=6.77 sys=0.25, real=4.32 secs]
2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-preclean-start]
2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew:
235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K) icms_dc=56
, 0.0182051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2010-08-26T15:44:48.108+0200: 38685.790: [CMS-concurrent-preclean:
0.023/0.543 secs] [Times: user=0.55 sys=0.05, real=0.55 secs]
2010-08-26T15:44:48.108+0200: 38685.790:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:44:48.483+0200: 38686.158: [CMS-concurrent-abortable-preclean:
0.040/0.368 secs] [Times: user=0.41 sys=0.02, real=0.38 secs]
2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856 K (235712
K)]38686.159: [Rescan (non-parallel) 38686.159: [grey object rescan,
0.0041031 secs]38686.163: [root rescan, 0.1153851 secs], 0.1195876
secs]38686.279: [weak refs processing, 0.0012088 secs]38686.280: [class
unloading, 0.0143079 secs]38686.294: [scrub symbol & string tables,
0.0159790 secs] [1 CMS-remark: 931150K(943744K)] 1068006K(1179456K),
0.1563910 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew:
235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K) icms_dc=56
, 0.0185256 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep: 0.470/0.992
secs] [Times: user=1.47 sys=0.02, real=1.00 secs]
2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset: 0.005/0.005
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew:
235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K) icms_dc=42
, 0.0179840 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark:
923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times: user=0.05
sys=0.00, real=0.05 secs]
2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew:
235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K) icms_dc=42
, 0.0179821 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew:
235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K) icms_dc=42
, 0.0182661 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew:
235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K) icms_dc=42
, 0.0269020 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark: 1.894/3.870
secs] [Times: user=5.74 sys=0.13, real=3.87 secs]
2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-preclean-start]
2010-08-26T15:44:54.551+0200: 38692.222: [CMS-concurrent-preclean:
0.015/0.020 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
2010-08-26T15:44:54.551+0200: 38692.222:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew:
235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K) icms_dc=42
, 0.0187843 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:55.581+0200: 38693.250: [CMS-concurrent-abortable-preclean:
0.050/1.028 secs] [Times: user=1.05 sys=0.01, real=1.03 secs]
2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686 K (235712
K)]38693.251: [Rescan (non-parallel) 38693.251: [grey object rescan,
0.0043953 secs]38693.255: [root rescan, 0.1209735 secs], 0.1254680
secs]38693.376: [weak refs processing, 0.0013433 secs]38693.378: [class
unloading, 0.0142815 secs]38693.392: [scrub symbol & string tables,
0.0160091 secs] [1 CMS-remark: 926149K(943744K)] 1066835K(1179456K),
0.1624119 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew:
235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K) icms_dc=42
, 0.0189967 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep: 0.474/1.149
secs] [Times: user=1.64 sys=0.02, real=1.15 secs]
2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset: 0.006/0.006
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew:
235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K) icms_dc=32
, 0.0189332 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark:
921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew:
235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K) icms_dc=32
, 0.0185097 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew:
235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K) icms_dc=32
, 0.0197977 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew:
235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K) icms_dc=32
, 0.0189818 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew:
235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K) icms_dc=32
, 0.0181892 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew:
235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K) icms_dc=32
, 0.0188287 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark: 1.899/6.275
secs] [Times: user=8.16 sys=0.06, real=6.27 secs]
2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-preclean-start]
2010-08-26T15:45:04.083+0200: 38701.756: [CMS-concurrent-preclean:
0.016/0.021 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
2010-08-26T15:45:04.083+0200: 38701.756:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew:
235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K) icms_dc=32
, 0.0185851 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:45:05.409+0200: 38703.087: [CMS-concurrent-abortable-preclean:
0.063/1.331 secs] [Times: user=1.39 sys=0.03, real=1.33 secs]
2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725 K (235712
K)]38703.088: [Rescan (non-parallel) 38703.088: [grey object rescan,
0.0041240 secs]38703.093: [root rescan, 0.1200361 secs], 0.1242579
secs]38703.213: [weak refs processing, 0.0010203 secs]38703.214: [class
unloading, 0.0142923 secs]38703.228: [scrub symbol & string tables,
0.0170438 secs] [1 CMS-remark: 924427K(943744K)] 1066152K(1179456K),
0.1618864 secs] [Times: user=0.17 sys=0.00, real=0.17 secs]
2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew:
235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K) icms_dc=32
, 0.0427672 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew:
235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K) icms_dc=32
, 0.0181045 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep: 0.472/2.032
secs] [Times: user=2.53 sys=0.08, real=2.03 secs]
2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset: 0.005/0.005
secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew:
235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K) icms_dc=24
, 0.0182417 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark:
926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times: user=0.06
sys=0.00, real=0.06 secs]
2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew:
235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K) icms_dc=39
, 0.0479217 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew:
235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K) icms_dc=54
, 0.0373840 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew:
235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K) icms_dc=69
, 0.0187606 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew (promotion
failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901:
[CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark:
1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs]
 (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs]
1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] icms_dc=86
, 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]
2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark:
933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times: user=0.01
sys=0.00, real=0.02 secs]
2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew:
235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K)
icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark: 1.964/2.114
secs] [Times: user=4.57 sys=0.11, real=2.12 secs]
2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-preclean-start]
2010-08-26T15:45:20.525+0200: 38718.202: [CMS-concurrent-preclean:
0.015/0.020 secs] [Times: user=0.02 sys=0.02, real=0.02 secs]
2010-08-26T15:45:20.525+0200: 38718.202:
[CMS-concurrent-abortable-preclean-start]
2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew:
235520K->235520K(235712K), 0.0000592 secs]38718.222:
[CMS2010-08-26T15:45:20.540+0200: 38718.223:
[CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times: user=0.02
sys=0.00, real=0.01 secs]
 (concurrent mode failure): 942749K->936038K(943744K), 4.0650806 secs]
1178269K->936038K(1179456K), [CMS Perm : 97371K->97371K(131072K)]
icms_dc=100 , 4.0653451 secs] [Times: user=4.07 sys=0.00, real=4.07 secs]
2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark:
936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew:
235520K->235520K(235712K), 0.0000588 secs]38723.817:
[CMS2010-08-26T15:45:27.030+0200: 38724.709: [CMS-concurrent-mark:
1.935/2.418 secs] [Times: user=3.65 sys=0.19, real=2.42 secs]
 (concurrent mode failure): 936038K->942227K(943744K), 4.9718511 secs]
1171558K->942227K(1179456K), [CMS Perm : 97393K->97393K(131072K)]
icms_dc=100 , 4.9721115 secs] [Times: user=4.98 sys=0.00, real=4.98 secs]
2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark:
942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew:
235520K->235520K(235712K), 0.0000623 secs]38729.759:
[CMS2010-08-26T15:45:33.051+0200: 38730.736: [CMS-concurrent-mark:
1.934/1.943 secs] [Times: user=3.28 sys=0.19, real=1.93 secs]
 (concurrent mode failure): 942227K->943743K(943744K), 4.9442127 secs]
1177747K->948024K(1179456K), [CMS Perm : 97396K->97396K(131072K)]
icms_dc=100 , 4.9446265 secs] [Times: user=4.91 sys=0.02, real=4.94 secs]
2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark:
943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419:
[CMS2010-08-26T15:45:39.463+0200: 38737.138: [CMS-concurrent-mark:
1.952/2.427 secs] [Times: user=3.95 sys=0.16, real=2.43 secs]
 (concurrent mode failure): 943743K->943743K(943744K), 4.7856092 secs]
1179455K->949021K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
icms_dc=100 , 4.7857927 secs] [Times: user=4.79 sys=0.00, real=4.79 secs]
2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark:
943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136:
[CMS2010-08-26T15:45:45.484+0200: 38743.168: [CMS-concurrent-mark:
1.952/1.954 secs] [Times: user=3.26 sys=0.05, real=1.95 secs]
 (concurrent mode failure): 943743K->943743K(943744K), 4.9692415 secs]
1179440K->949759K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
icms_dc=100 , 4.9694250 secs] [Times: user=4.96 sys=0.02, real=4.98 secs]
2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark:
943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times: user=0.02
sys=0.00, real=0.02 secs]
2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634:
[CMS2010-08-26T15:45:51.833+0200: 38749.515: [CMS-concurrent-mark:
1.927/1.938 secs] [Times: user=3.23 sys=0.11, real=1.93 secs]
 (concurrent mode failure): 943743K->937800K(943744K), 4.9818453 secs]
1179455K->937800K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
icms_dc=100 , 4.9820296 secs] [Times: user=4.95 sys=0.02, real=4.98 secs]
2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark:
937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew:
235520K->235520K(235712K), 0.0000597 secs]38754.631:
[CMS2010-08-26T15:45:57.995+0200: 38755.679: [CMS-concurrent-mark:
2.057/2.059 secs] [Times: user=3.29 sys=0.23, real=2.06 secs]
 (concurrent mode failure): 937800K->936173K(943744K), 5.1274815 secs]
1173320K->936173K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
icms_dc=100 , 5.1277524 secs] [Times: user=5.13 sys=0.00, real=5.13 secs]
2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark:
936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times: user=0.00
sys=0.00, real=0.00 secs]
2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew (promotion
failed): 235520K->235520K(235712K), 0.1302956 secs]38761.420:
[CMS2010-08-26T15:46:04.609+0200: 38762.289: [CMS-concurrent-mark:
1.938/2.071 secs] [Times: user=3.54 sys=0.11, real=2.07 secs]
 (concurrent mode failure): 943229K->939271K(943744K), 4.9629545 secs]
1171693K->939271K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
icms_dc=100 , 5.0934938 secs] [Times: user=5.09 sys=0.00, real=5.10 secs]

Thanks again for any help,

Adam

--
Adam Hawthorne
Software Engineer
BASIS International Ltd.
www.basis.com
+1.505.345.5232 Phone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100831/16a5b52f/attachment-0001.html 
-------------- next part --------------
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From jon.masamitsu at oracle.com  Tue Aug 31 13:54:35 2010
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Tue, 31 Aug 2010 13:54:35 -0700
Subject: iCMS question
In-Reply-To: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
References: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
Message-ID: <4C7D6C0B.9070103@oracle.com>

I think the short answer is that your heap is full.

This is a minor collection that turns into a full collection (so you
don't see the telltale "Full GC" message).

2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew 
(promotion failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901: 
[CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark: 
1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs] 
 (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs] 
1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] 
icms_dc=86 , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]

After the full collection the heap is still mostly full.

943043K->933853K(943744K)

I think your advice to increase the size of the should will help.


On 08/31/10 12:36, Adam Hawthorne wrote:
> Hi all,
>
> I'm looking at an i-cms log, and I don't understand why I'm seeing 
> what I'm seeing.  Any help is appreciated.  At about  38718.035 in the 
> log below, icms_dc begins to report 100, and just after that I see 
> several back-to-back concurrent mode failures.  During that time, the 
> ParNew collections are reporting that they're doing no work.  This 
> happens on and off for about 20 minutes in the log.  Obviously the 
> collections are taking seconds, but I don't see a Full GC every time. 
>  What is it doing when it's not doing a full GC?  Is the continual CMS 
> collection preventing the young generation from being collected (or 
> perhaps collecting it simultaneously)?  When icms_dc=100, does that 
> imply the application is stopped?
>
> There was some uninformed command line twiddling going on here, so the 
> GC parameters were:
> -Xmx1152m
> -Xms256m
> -XX:NewRatio=4
> -XX:MaxPermSize=128m
> -XX:+UseParNewGC
> -XX:-CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled
> -XX:+UseCMSCompactAtFullCollection
> -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSClassUnloadingEnabled
> -XX:+CMSIncrementalMode
> -XX:+CMSIncrementalPacing
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:ParallelGCThreads=1
> -XX:MaxTenuringThreshold=0
> -XX:SurvivorRatio=1024
> -XX:SoftRefLRUPolicyMSPerMB=1000
> -XX:+DisableExplicitGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps
>
> I had them remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0, 
> and -XX:SurvivorRatio=1024 , thinking that since the duty cycle was 
> approaching 100%, and because the ParNew pause times were so low, we 
> could probably afford to spend more time doing young gc's instead of 
> promoting.  I'm thinking I should probably tell them to increase their 
> -Xmx and decrease their NewRatio as well.  Is there anything else?  I 
> haven't done much tuning of i-cms in general, but the standard i-cms 
> safety options don't seem like they would help here.
>
> Here's the log excerpt:
>
> 2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew: 
> 235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K) 
> icms_dc=41 , 0.0189305 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark: 
> 923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times: user=0.06 
> sys=0.00, real=0.06 secs] 
> 2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew: 
> 235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K) 
> icms_dc=41 , 0.0197785 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew: 
> 235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K) 
> icms_dc=41 , 0.0201448 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew: 
> 235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K) 
> icms_dc=56 , 0.0551800 secs] [Times: user=0.05 sys=0.00, real=0.05 secs] 
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark: 
> 1.903/4.332 secs] [Times: user=6.77 sys=0.25, real=4.32 secs] 
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew: 
> 235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K) 
> icms_dc=56 , 0.0182051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
> 2010-08-26T15:44:48.108+0200: 38685.790: [CMS-concurrent-preclean: 
> 0.023/0.543 secs] [Times: user=0.55 sys=0.05, real=0.55 secs] 
> 2010-08-26T15:44:48.108+0200: 38685.790: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:48.483+0200: 38686.158: 
> [CMS-concurrent-abortable-preclean: 0.040/0.368 secs] [Times: 
> user=0.41 sys=0.02, real=0.38 secs] 
> 2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856 K 
> (235712 K)]38686.159: [Rescan (non-parallel) 38686.159: [grey object 
> rescan, 0.0041031 secs]38686.163: [root rescan, 0.1153851 secs], 
> 0.1195876 secs]38686.279: [weak refs processing, 0.0012088 
> secs]38686.280: [class unloading, 0.0143079 secs]38686.294: [scrub 
> symbol & string tables, 0.0159790 secs] [1 CMS-remark: 
> 931150K(943744K)] 1068006K(1179456K), 0.1563910 secs] [Times: 
> user=0.16 sys=0.00, real=0.16 secs] 
> 2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew: 
> 235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K) 
> icms_dc=56 , 0.0185256 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep: 
> 0.470/0.992 secs] [Times: user=1.47 sys=0.02, real=1.00 secs] 
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset: 
> 0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
> 2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew: 
> 235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K) 
> icms_dc=42 , 0.0179840 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark: 
> 923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times: user=0.05 
> sys=0.00, real=0.05 secs] 
> 2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew: 
> 235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K) 
> icms_dc=42 , 0.0179821 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew: 
> 235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K) 
> icms_dc=42 , 0.0182661 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew: 
> 235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K) 
> icms_dc=42 , 0.0269020 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark: 
> 1.894/3.870 secs] [Times: user=5.74 sys=0.13, real=3.87 secs] 
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:54.551+0200: 38692.222: [CMS-concurrent-preclean: 
> 0.015/0.020 secs] [Times: user=0.05 sys=0.02, real=0.03 secs] 
> 2010-08-26T15:44:54.551+0200: 38692.222: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew: 
> 235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K) 
> icms_dc=42 , 0.0187843 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:55.581+0200: 38693.250: 
> [CMS-concurrent-abortable-preclean: 0.050/1.028 secs] [Times: 
> user=1.05 sys=0.01, real=1.03 secs] 
> 2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686 K 
> (235712 K)]38693.251: [Rescan (non-parallel) 38693.251: [grey object 
> rescan, 0.0043953 secs]38693.255: [root rescan, 0.1209735 secs], 
> 0.1254680 secs]38693.376: [weak refs processing, 0.0013433 
> secs]38693.378: [class unloading, 0.0142815 secs]38693.392: [scrub 
> symbol & string tables, 0.0160091 secs] [1 CMS-remark: 
> 926149K(943744K)] 1066835K(1179456K), 0.1624119 secs] [Times: 
> user=0.16 sys=0.00, real=0.16 secs] 
> 2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew: 
> 235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K) 
> icms_dc=42 , 0.0189967 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep: 
> 0.474/1.149 secs] [Times: user=1.64 sys=0.02, real=1.15 secs] 
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset: 
> 0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
> 2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew: 
> 235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K) 
> icms_dc=32 , 0.0189332 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark: 
> 921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times: 
> user=0.06 sys=0.00, real=0.06 secs] 
> 2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew: 
> 235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K) 
> icms_dc=32 , 0.0185097 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
> 2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew: 
> 235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K) 
> icms_dc=32 , 0.0197977 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew: 
> 235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K) 
> icms_dc=32 , 0.0189818 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew: 
> 235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K) 
> icms_dc=32 , 0.0181892 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
> 2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew: 
> 235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K) 
> icms_dc=32 , 0.0188287 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark: 
> 1.899/6.275 secs] [Times: user=8.16 sys=0.06, real=6.27 secs] 
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:04.083+0200: 38701.756: [CMS-concurrent-preclean: 
> 0.016/0.021 secs] [Times: user=0.05 sys=0.02, real=0.03 secs] 
> 2010-08-26T15:45:04.083+0200: 38701.756: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew: 
> 235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K) 
> icms_dc=32 , 0.0185851 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:05.409+0200: 38703.087: 
> [CMS-concurrent-abortable-preclean: 0.063/1.331 secs] [Times: 
> user=1.39 sys=0.03, real=1.33 secs] 
> 2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725 K 
> (235712 K)]38703.088: [Rescan (non-parallel) 38703.088: [grey object 
> rescan, 0.0041240 secs]38703.093: [root rescan, 0.1200361 secs], 
> 0.1242579 secs]38703.213: [weak refs processing, 0.0010203 
> secs]38703.214: [class unloading, 0.0142923 secs]38703.228: [scrub 
> symbol & string tables, 0.0170438 secs] [1 CMS-remark: 
> 924427K(943744K)] 1066152K(1179456K), 0.1618864 secs] [Times: 
> user=0.17 sys=0.00, real=0.17 secs] 
> 2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
> 2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew: 
> 235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K) 
> icms_dc=32 , 0.0427672 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew: 
> 235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K) 
> icms_dc=32 , 0.0181045 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep: 
> 0.472/2.032 secs] [Times: user=2.53 sys=0.08, real=2.03 secs] 
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
> 2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset: 
> 0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew: 
> 235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K) 
> icms_dc=24 , 0.0182417 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark: 
> 926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times: 
> user=0.06 sys=0.00, real=0.06 secs] 
> 2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew: 
> 235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K) 
> icms_dc=39 , 0.0479217 secs] [Times: user=0.05 sys=0.00, real=0.05 secs] 
> 2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew: 
> 235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K) 
> icms_dc=54 , 0.0373840 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew: 
> 235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K) 
> icms_dc=69 , 0.0187606 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
> 2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew 
> (promotion failed): 235520K->235520K(235712K), 1.0952192 
> secs]38711.901: [CMS2010-08-26T15:45:14.222+0200: 38711.903: 
> [CMS-concurrent-mark: 1.934/5.410 secs] [Times: user=8.20 sys=0.25, 
> real=5.41 secs] 
>  (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs] 
> 1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] 
> icms_dc=86 , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs] 
> 2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark: 
> 933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times: user=0.01 
> sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew: 
> 235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K) 
> icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00, real=0.06 secs] 
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark: 
> 1.964/2.114 secs] [Times: user=4.57 sys=0.11, real=2.12 secs] 
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:20.525+0200: 38718.202: [CMS-concurrent-preclean: 
> 0.015/0.020 secs] [Times: user=0.02 sys=0.02, real=0.02 secs] 
> 2010-08-26T15:45:20.525+0200: 38718.202: 
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew: 
> 235520K->235520K(235712K), 0.0000592 secs]38718.222: 
> [CMS2010-08-26T15:45:20.540+0200: 38718.223: 
> [CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times: 
> user=0.02 sys=0.00, real=0.01 secs] 
>  (concurrent mode failure): 942749K->936038K(943744K), 4.0650806 secs] 
> 1178269K->936038K(1179456K), [CMS Perm : 97371K->97371K(131072K)] 
> icms_dc=100 , 4.0653451 secs] [Times: user=4.07 sys=0.00, real=4.07 secs] 
> 2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark: 
> 936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew: 
> 235520K->235520K(235712K), 0.0000588 secs]38723.817: 
> [CMS2010-08-26T15:45:27.030+0200: 38724.709: [CMS-concurrent-mark: 
> 1.935/2.418 secs] [Times: user=3.65 sys=0.19, real=2.42 secs] 
>  (concurrent mode failure): 936038K->942227K(943744K), 4.9718511 secs] 
> 1171558K->942227K(1179456K), [CMS Perm : 97393K->97393K(131072K)] 
> icms_dc=100 , 4.9721115 secs] [Times: user=4.98 sys=0.00, real=4.98 secs] 
> 2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark: 
> 942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew: 
> 235520K->235520K(235712K), 0.0000623 secs]38729.759: 
> [CMS2010-08-26T15:45:33.051+0200: 38730.736: [CMS-concurrent-mark: 
> 1.934/1.943 secs] [Times: user=3.28 sys=0.19, real=1.93 secs] 
>  (concurrent mode failure): 942227K->943743K(943744K), 4.9442127 secs] 
> 1177747K->948024K(1179456K), [CMS Perm : 97396K->97396K(131072K)] 
> icms_dc=100 , 4.9446265 secs] [Times: user=4.91 sys=0.02, real=4.94 secs] 
> 2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark: 
> 943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419: 
> [CMS2010-08-26T15:45:39.463+0200: 38737.138: [CMS-concurrent-mark: 
> 1.952/2.427 secs] [Times: user=3.95 sys=0.16, real=2.43 secs] 
>  (concurrent mode failure): 943743K->943743K(943744K), 4.7856092 secs] 
> 1179455K->949021K(1179456K), [CMS Perm : 97406K->97406K(131072K)] 
> icms_dc=100 , 4.7857927 secs] [Times: user=4.79 sys=0.00, real=4.79 secs] 
> 2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark: 
> 943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136: 
> [CMS2010-08-26T15:45:45.484+0200: 38743.168: [CMS-concurrent-mark: 
> 1.952/1.954 secs] [Times: user=3.26 sys=0.05, real=1.95 secs] 
>  (concurrent mode failure): 943743K->943743K(943744K), 4.9692415 secs] 
> 1179440K->949759K(1179456K), [CMS Perm : 97406K->97406K(131072K)] 
> icms_dc=100 , 4.9694250 secs] [Times: user=4.96 sys=0.02, real=4.98 secs] 
> 2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark: 
> 943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times: user=0.02 
> sys=0.00, real=0.02 secs] 
> 2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634: 
> [CMS2010-08-26T15:45:51.833+0200: 38749.515: [CMS-concurrent-mark: 
> 1.927/1.938 secs] [Times: user=3.23 sys=0.11, real=1.93 secs] 
>  (concurrent mode failure): 943743K->937800K(943744K), 4.9818453 secs] 
> 1179455K->937800K(1179456K), [CMS Perm : 97407K->97407K(131072K)] 
> icms_dc=100 , 4.9820296 secs] [Times: user=4.95 sys=0.02, real=4.98 secs] 
> 2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark: 
> 937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew: 
> 235520K->235520K(235712K), 0.0000597 secs]38754.631: 
> [CMS2010-08-26T15:45:57.995+0200: 38755.679: [CMS-concurrent-mark: 
> 2.057/2.059 secs] [Times: user=3.29 sys=0.23, real=2.06 secs] 
>  (concurrent mode failure): 937800K->936173K(943744K), 5.1274815 secs] 
> 1173320K->936173K(1179456K), [CMS Perm : 97407K->97407K(131072K)] 
> icms_dc=100 , 5.1277524 secs] [Times: user=5.13 sys=0.00, real=5.13 secs] 
> 2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark: 
> 936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times: user=0.00 
> sys=0.00, real=0.00 secs] 
> 2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
> 2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew 
> (promotion failed): 235520K->235520K(235712K), 0.1302956 
> secs]38761.420: [CMS2010-08-26T15:46:04.609+0200: 38762.289: 
> [CMS-concurrent-mark: 1.938/2.071 secs] [Times: user=3.54 sys=0.11, 
> real=2.07 secs] 
>  (concurrent mode failure): 943229K->939271K(943744K), 4.9629545 secs] 
> 1171693K->939271K(1179456K), [CMS Perm : 97407K->97407K(131072K)] 
> icms_dc=100 , 5.0934938 secs] [Times: user=5.09 sys=0.00, real=5.10 secs] 
>
> Thanks again for any help,
>
> Adam
>
> --
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com <http://www.basis.com>
> +1.505.345.5232 Phone
> ------------------------------------------------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100831/04c8eeeb/attachment-0001.html 
-------------- next part --------------
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From adamh at basis.com  Tue Aug 31 14:05:12 2010
From: adamh at basis.com (Adam Hawthorne)
Date: Tue, 31 Aug 2010 17:05:12 -0400
Subject: iCMS question
In-Reply-To: <4C7D6C0B.9070103@oracle.com>
References: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
	<4C7D6C0B.9070103@oracle.com>
Message-ID: <AANLkTinGk8S7hZ6OKq-B3Y+dqypWYJA+D+1quhbQaVXi@mail.gmail.com>

Thanks, somehow I missed the obvious answer.

Adam

--
Adam Hawthorne
Software Engineer
BASIS International Ltd.
www.basis.com
+1.505.345.5232 Phone


On Tue, Aug 31, 2010 at 16:54, Jon Masamitsu <jon.masamitsu at oracle.com>wrote:

>  I think the short answer is that your heap is full.
>
> This is a minor collection that turns into a full collection (so you
> don't see the telltale "Full GC" message).
>
>
> 2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew (promotion
> failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901:
> [CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark:
> 1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs]
>  (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs]
> 1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] icms_dc=86
> , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]
>
> After the full collection the heap is still mostly full.
>
> 943043K->933853K(943744K)
>
> I think your advice to increase the size of the should will help.
>
>
> On 08/31/10 12:36, Adam Hawthorne wrote:
>
> Hi all,
>
>  I'm looking at an i-cms log, and I don't understand why I'm seeing what
> I'm seeing.  Any help is appreciated.  At about  38718.035 in the log below,
> icms_dc begins to report 100, and just after that I see several back-to-back
> concurrent mode failures.  During that time, the ParNew collections are
> reporting that they're doing no work.  This happens on and off for about 20
> minutes in the log.  Obviously the collections are taking seconds, but I
> don't see a Full GC every time.  What is it doing when it's not doing a full
> GC?  Is the continual CMS collection preventing the young generation from
> being collected (or perhaps collecting it simultaneously)?  When
> icms_dc=100, does that imply the application is stopped?
>
>  There was some uninformed command line twiddling going on here, so the GC
> parameters were:
> -Xmx1152m
> -Xms256m
> -XX:NewRatio=4
> -XX:MaxPermSize=128m
> -XX:+UseParNewGC
> -XX:-CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled
> -XX:+UseCMSCompactAtFullCollection
> -XX:CMSFullGCsBeforeCompaction=1
> -XX:+CMSClassUnloadingEnabled
> -XX:+CMSIncrementalMode
> -XX:+CMSIncrementalPacing
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:ParallelGCThreads=1
> -XX:MaxTenuringThreshold=0
> -XX:SurvivorRatio=1024
> -XX:SoftRefLRUPolicyMSPerMB=1000
> -XX:+DisableExplicitGC
> -XX:+PrintGCDetails
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps
>
>  I had them remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0,
> and -XX:SurvivorRatio=1024 , thinking that since the duty cycle was
> approaching 100%, and because the ParNew pause times were so low, we could
> probably afford to spend more time doing young gc's instead of promoting.
>  I'm thinking I should probably tell them to increase their -Xmx and
> decrease their NewRatio as well.  Is there anything else?  I haven't done
> much tuning of i-cms in general, but the standard i-cms safety options don't
> seem like they would help here.
>
>  Here's the log excerpt:
>
>  2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew:
> 235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K) icms_dc=41
> , 0.0189305 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark:
> 923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times: user=0.06
> sys=0.00, real=0.06 secs]
> 2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew:
> 235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K) icms_dc=41
> , 0.0197785 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew:
> 235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K) icms_dc=41
> , 0.0201448 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew:
> 235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K) icms_dc=56
> , 0.0551800 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark: 1.903/4.332
> secs] [Times: user=6.77 sys=0.25, real=4.32 secs]
> 2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew:
> 235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K) icms_dc=56
> , 0.0182051 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2010-08-26T15:44:48.108+0200: 38685.790: [CMS-concurrent-preclean:
> 0.023/0.543 secs] [Times: user=0.55 sys=0.05, real=0.55 secs]
> 2010-08-26T15:44:48.108+0200: 38685.790:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:48.483+0200: 38686.158:
> [CMS-concurrent-abortable-preclean: 0.040/0.368 secs] [Times: user=0.41
> sys=0.02, real=0.38 secs]
> 2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856 K (235712
> K)]38686.159: [Rescan (non-parallel) 38686.159: [grey object rescan,
> 0.0041031 secs]38686.163: [root rescan, 0.1153851 secs], 0.1195876
> secs]38686.279: [weak refs processing, 0.0012088 secs]38686.280: [class
> unloading, 0.0143079 secs]38686.294: [scrub symbol & string tables,
> 0.0159790 secs] [1 CMS-remark: 931150K(943744K)] 1068006K(1179456K),
> 0.1563910 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
> 2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew:
> 235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K) icms_dc=56
> , 0.0185256 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep: 0.470/0.992
> secs] [Times: user=1.47 sys=0.02, real=1.00 secs]
> 2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset: 0.005/0.005
> secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew:
> 235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K) icms_dc=42
> , 0.0179840 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark:
> 923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times: user=0.05
> sys=0.00, real=0.05 secs]
> 2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew:
> 235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K) icms_dc=42
> , 0.0179821 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew:
> 235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K) icms_dc=42
> , 0.0182661 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew:
> 235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K) icms_dc=42
> , 0.0269020 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark: 1.894/3.870
> secs] [Times: user=5.74 sys=0.13, real=3.87 secs]
> 2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-preclean-start]
> 2010-08-26T15:44:54.551+0200: 38692.222: [CMS-concurrent-preclean:
> 0.015/0.020 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
> 2010-08-26T15:44:54.551+0200: 38692.222:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew:
> 235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K) icms_dc=42
> , 0.0187843 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:55.581+0200: 38693.250:
> [CMS-concurrent-abortable-preclean: 0.050/1.028 secs] [Times: user=1.05
> sys=0.01, real=1.03 secs]
> 2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686 K (235712
> K)]38693.251: [Rescan (non-parallel) 38693.251: [grey object rescan,
> 0.0043953 secs]38693.255: [root rescan, 0.1209735 secs], 0.1254680
> secs]38693.376: [weak refs processing, 0.0013433 secs]38693.378: [class
> unloading, 0.0142815 secs]38693.392: [scrub symbol & string tables,
> 0.0160091 secs] [1 CMS-remark: 926149K(943744K)] 1066835K(1179456K),
> 0.1624119 secs] [Times: user=0.16 sys=0.00, real=0.16 secs]
> 2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
> 2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew:
> 235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K) icms_dc=42
> , 0.0189967 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep: 0.474/1.149
> secs] [Times: user=1.64 sys=0.02, real=1.15 secs]
> 2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
> 2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset: 0.006/0.006
> secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew:
> 235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K) icms_dc=32
> , 0.0189332 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark:
> 921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times: user=0.06
> sys=0.00, real=0.06 secs]
> 2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
> 2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew:
> 235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K) icms_dc=32
> , 0.0185097 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
> 2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew:
> 235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K) icms_dc=32
> , 0.0197977 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew:
> 235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K) icms_dc=32
> , 0.0189818 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew:
> 235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K) icms_dc=32
> , 0.0181892 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
> 2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew:
> 235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K) icms_dc=32
> , 0.0188287 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark: 1.899/6.275
> secs] [Times: user=8.16 sys=0.06, real=6.27 secs]
> 2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:04.083+0200: 38701.756: [CMS-concurrent-preclean:
> 0.016/0.021 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]
> 2010-08-26T15:45:04.083+0200: 38701.756:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew:
> 235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K) icms_dc=32
> , 0.0185851 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:05.409+0200: 38703.087:
> [CMS-concurrent-abortable-preclean: 0.063/1.331 secs] [Times: user=1.39
> sys=0.03, real=1.33 secs]
> 2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725 K (235712
> K)]38703.088: [Rescan (non-parallel) 38703.088: [grey object rescan,
> 0.0041240 secs]38703.093: [root rescan, 0.1200361 secs], 0.1242579
> secs]38703.213: [weak refs processing, 0.0010203 secs]38703.214: [class
> unloading, 0.0142923 secs]38703.228: [scrub symbol & string tables,
> 0.0170438 secs] [1 CMS-remark: 924427K(943744K)] 1066152K(1179456K),
> 0.1618864 secs] [Times: user=0.17 sys=0.00, real=0.17 secs]
> 2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
> 2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew:
> 235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K) icms_dc=32
> , 0.0427672 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew:
> 235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K) icms_dc=32
> , 0.0181045 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep: 0.472/2.032
> secs] [Times: user=2.53 sys=0.08, real=2.03 secs]
> 2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
> 2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset: 0.005/0.005
> secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew:
> 235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K) icms_dc=24
> , 0.0182417 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark:
> 926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times: user=0.06
> sys=0.00, real=0.06 secs]
> 2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew:
> 235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K) icms_dc=39
> , 0.0479217 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
> 2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew:
> 235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K) icms_dc=54
> , 0.0373840 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew:
> 235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K) icms_dc=69
> , 0.0187606 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
> 2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew (promotion
> failed): 235520K->235520K(235712K), 1.0952192 secs]38711.901:
> [CMS2010-08-26T15:45:14.222+0200: 38711.903: [CMS-concurrent-mark:
> 1.934/5.410 secs] [Times: user=8.20 sys=0.25, real=5.41 secs]
>  (concurrent mode failure): 943043K->933853K(943744K), 4.0906398 secs]
> 1172473K->933853K(1179456K), [CMS Perm : 97361K->97361K(131072K)] icms_dc=86
> , 5.1861184 secs] [Times: user=5.18 sys=0.00, real=5.18 secs]
> 2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark:
> 933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times: user=0.01
> sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew:
> 235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K)
> icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark: 1.964/2.114
> secs] [Times: user=4.57 sys=0.11, real=2.12 secs]
> 2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-preclean-start]
> 2010-08-26T15:45:20.525+0200: 38718.202: [CMS-concurrent-preclean:
> 0.015/0.020 secs] [Times: user=0.02 sys=0.02, real=0.02 secs]
> 2010-08-26T15:45:20.525+0200: 38718.202:
> [CMS-concurrent-abortable-preclean-start]
> 2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew:
> 235520K->235520K(235712K), 0.0000592 secs]38718.222:
> [CMS2010-08-26T15:45:20.540+0200: 38718.223:
> [CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times: user=0.02
> sys=0.00, real=0.01 secs]
>  (concurrent mode failure): 942749K->936038K(943744K), 4.0650806 secs]
> 1178269K->936038K(1179456K), [CMS Perm : 97371K->97371K(131072K)]
> icms_dc=100 , 4.0653451 secs] [Times: user=4.07 sys=0.00, real=4.07 secs]
> 2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark:
> 936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew:
> 235520K->235520K(235712K), 0.0000588 secs]38723.817:
> [CMS2010-08-26T15:45:27.030+0200: 38724.709: [CMS-concurrent-mark:
> 1.935/2.418 secs] [Times: user=3.65 sys=0.19, real=2.42 secs]
>  (concurrent mode failure): 936038K->942227K(943744K), 4.9718511 secs]
> 1171558K->942227K(1179456K), [CMS Perm : 97393K->97393K(131072K)]
> icms_dc=100 , 4.9721115 secs] [Times: user=4.98 sys=0.00, real=4.98 secs]
> 2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark:
> 942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew:
> 235520K->235520K(235712K), 0.0000623 secs]38729.759:
> [CMS2010-08-26T15:45:33.051+0200: 38730.736: [CMS-concurrent-mark:
> 1.934/1.943 secs] [Times: user=3.28 sys=0.19, real=1.93 secs]
>  (concurrent mode failure): 942227K->943743K(943744K), 4.9442127 secs]
> 1177747K->948024K(1179456K), [CMS Perm : 97396K->97396K(131072K)]
> icms_dc=100 , 4.9446265 secs] [Times: user=4.91 sys=0.02, real=4.94 secs]
> 2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark:
> 943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419:
> [CMS2010-08-26T15:45:39.463+0200: 38737.138: [CMS-concurrent-mark:
> 1.952/2.427 secs] [Times: user=3.95 sys=0.16, real=2.43 secs]
>  (concurrent mode failure): 943743K->943743K(943744K), 4.7856092 secs]
> 1179455K->949021K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
> icms_dc=100 , 4.7857927 secs] [Times: user=4.79 sys=0.00, real=4.79 secs]
> 2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark:
> 943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136:
> [CMS2010-08-26T15:45:45.484+0200: 38743.168: [CMS-concurrent-mark:
> 1.952/1.954 secs] [Times: user=3.26 sys=0.05, real=1.95 secs]
>  (concurrent mode failure): 943743K->943743K(943744K), 4.9692415 secs]
> 1179440K->949759K(1179456K), [CMS Perm : 97406K->97406K(131072K)]
> icms_dc=100 , 4.9694250 secs] [Times: user=4.96 sys=0.02, real=4.98 secs]
> 2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark:
> 943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times: user=0.02
> sys=0.00, real=0.02 secs]
> 2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634:
> [CMS2010-08-26T15:45:51.833+0200: 38749.515: [CMS-concurrent-mark:
> 1.927/1.938 secs] [Times: user=3.23 sys=0.11, real=1.93 secs]
>  (concurrent mode failure): 943743K->937800K(943744K), 4.9818453 secs]
> 1179455K->937800K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
> icms_dc=100 , 4.9820296 secs] [Times: user=4.95 sys=0.02, real=4.98 secs]
> 2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark:
> 937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
> 2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew:
> 235520K->235520K(235712K), 0.0000597 secs]38754.631:
> [CMS2010-08-26T15:45:57.995+0200: 38755.679: [CMS-concurrent-mark:
> 2.057/2.059 secs] [Times: user=3.29 sys=0.23, real=2.06 secs]
>  (concurrent mode failure): 937800K->936173K(943744K), 5.1274815 secs]
> 1173320K->936173K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
> icms_dc=100 , 5.1277524 secs] [Times: user=5.13 sys=0.00, real=5.13 secs]
> 2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark:
> 936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times: user=0.00
> sys=0.00, real=0.00 secs]
> 2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
> 2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew (promotion
> failed): 235520K->235520K(235712K), 0.1302956 secs]38761.420:
> [CMS2010-08-26T15:46:04.609+0200: 38762.289: [CMS-concurrent-mark:
> 1.938/2.071 secs] [Times: user=3.54 sys=0.11, real=2.07 secs]
>  (concurrent mode failure): 943229K->939271K(943744K), 4.9629545 secs]
> 1171693K->939271K(1179456K), [CMS Perm : 97407K->97407K(131072K)]
> icms_dc=100 , 5.0934938 secs] [Times: user=5.09 sys=0.00, real=5.10 secs]
>
>  Thanks again for any help,
>
> Adam
>
> --
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com
> +1.505.345.5232 Phone
>
> ------------------------------
>
> _______________________________________________
> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100831/e3c298df/attachment-0001.html 
-------------- next part --------------
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From y.s.ramakrishna at oracle.com  Tue Aug 31 15:27:25 2010
From: y.s.ramakrishna at oracle.com (Y. S. Ramakrishna)
Date: Tue, 31 Aug 2010 15:27:25 -0700
Subject: Long GC pause time
In-Reply-To: <4C7D49E4.6060908@oracle.com>
References: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>
	<4C7D49E4.6060908@oracle.com>
Message-ID: <4C7D81CD.5050706@oracle.com>

Noam, also please provision and use the survivor spaces to age
objects and reduce pressure on CMS. The premature
promotion and subsequent death is probably exacerbating the
proliferating Ref object deaths (if i may be permitted an
oxymoron) in the old gen.

-- ramki

On 08/31/10 11:28, Jon Masamitsu wrote:
> Try -XX+ParallelRefProcEnabled
> 
> Your spending much of the pause time doing Reference processing
> 
> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 
> 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
> [Times: user=1.68 sys=0.01, real=1.41 secs]
> 
> (see the "[weak refs processing, 1.3860230 secs]") so doing it in 
> parallel might
> help.  ParallelRefProcEnabled isn't always a win so it's not "on" by 
> default.
> 
> You might also try -XX:+CMSScavengeBeforeRemark.  That's a just a guess.
> Sometimes doing a minor collection before the remark helps.
> 
> On 08/31/10 08:22, noam katzowicz wrote:
>> Hello.
>>  
>> We are running JBoss AS over RH5, using java 6u21, with the following 
>> GC params:
>>  
>> -Xms6750m -Xmx6750m
>> -XX:+UseConcMarkSweepGC
>> -XX:SurvivorRatio=128
>> -XX:MaxTenuringThreshold=0
>> -XX:+UseCompressedOops
>> -XX:CMSInitiatingOccupancyFraction=60
>> -Dsun.rmi.dgc.client.gcInterval=3600000
>> -Dsun.rmi.dgc.server.gcInterval=3600000
>> -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
>>  
>> We are experiencing relatively long GC pause time of over 1 second.
>> Our application allocates hundreds of MBs per second. We?ve tried 
>> different generation sizes but the above was the best we came up with. 
>> Your help is appreciated.
>>  
>>
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 
>> 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>> 89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 
>> sys=2.30, real=1.65 secs]
>> 89.487: [CMS-concurrent-abortable-preclean-start]
>> 89.804: [GC 89.804: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 
>> 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs]
>> 89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: 
>> user=2.42 sys=0.54, real=0.38 secs]
>> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) 
>> , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
>> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
>> [Times: user=1.68 sys=0.01, real=1.41 secs]
>> 91.284: [CMS-concurrent-sweep-start]
>> 91.590: [GC 91.591: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 
>> 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]
>> 91.860: [GC 91.860: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 
>> 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>> 92.786: [GC 92.787: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 
>> 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs]
>> 93.106: [GC 93.107: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 
>> 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]
>> 93.371: [GC 93.371: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 
>> 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
>> 94.637: [GC 94.638: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 
>> 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs]
>> 94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 
>> sys=3.23, real=3.58 secs]
>> 94.863: [CMS-concurrent-reset-start]
>> 94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 
>> sys=0.01, real=0.02 secs]
>> 95.981: [GC 95.981: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 
>> 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs]
>> 100.179: [GC 100.180: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 
>> 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs]
>> 100.909: [GC 100.909: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 
>> 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs]
>> 101.613: [GC 101.613: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 
>> 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>> 102.072: [GC 102.073: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 
>> 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs]
>> 102.756: [GC 102.756: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 
>> 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>> 103.452: [GC 103.453: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 
>> 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>> 104.462: [GC 104.462: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 
>> 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs]
>> 106.274: [GC 106.274: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 
>> 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs]
>> 108.154: [GC 108.155: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 
>> 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs]
>> 108.882: [GC 108.883: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 
>> 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs]
>> 109.482: [GC 109.482: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 
>> 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs]
>> 109.948: [GC 109.949: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 
>> 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs]
>> 110.035: [GC 110.035: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 
>> 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
>> 110.484: [GC 110.485: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 
>> 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs]
>> 110.910: [GC 110.910: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 
>> 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs]
>> 111.276: [GC 111.277: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 
>> 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs]
>> 111.658: [GC 111.659: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 
>> 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs]
>> 112.068: [GC 112.068: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 
>> 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs]
>> 112.497: [GC 112.498: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 
>> 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs]
>> 112.911: [GC 112.911: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 
>> 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs]
>> 113.362: [GC 113.363: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 
>> 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs]
>> 113.829: [GC 113.829: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 
>> 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs]
>> 114.311: [GC 114.311: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 
>> 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>> 114.841: [GC 114.842: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 
>> 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>> 115.356: [GC 115.357: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 
>> 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>> 115.829: [GC 115.830: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 
>> 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs]
>> 116.271: [GC 116.271: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 
>> 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs]
>> 116.716: [GC 116.717: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 
>> 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]
>> 117.145: [GC 117.146: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 
>> 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs]
>> 117.543: [GC 117.545: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 
>> 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs]
>> 117.956: [GC 117.956: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 
>> 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs]
>> 118.385: [GC 118.386: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 
>> 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>> 118.826: [GC 118.827: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 
>> 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs]
>> 119.233: [GC 119.234: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 
>> 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs]
>> 119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 
>> 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, 
>> real=0.01 secs]
>> 119.311: [CMS-concurrent-mark-start]
>> 119.699: [GC 119.699: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 
>> 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs]
>> 120.081: [GC 120.082: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 
>> 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>> 120.538: [GC 120.539: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 
>> 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs]
>> 120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 
>> sys=1.73, real=1.31 secs]
>> 120.621: [CMS-concurrent-preclean-start]
>> 120.942: [GC 120.943: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 
>> 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs]
>> 121.337: [GC 121.338: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 
>> 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs]
>> 121.766: [GC 121.767: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 
>> 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>> 122.174: [GC 122.175: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 
>> 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs]
>> 122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: 
>> user=10.25 sys=2.21, real=1.63 secs]
>> 122.257: [CMS-concurrent-abortable-preclean-start]
>> 122.631: [GC 122.631: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 
>> 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs]
>> 122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: 
>> user=2.50 sys=0.52, real=0.44 secs]
>> 122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan 
>> (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 
>> secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 
>> secs] [Times: user=1.42 sys=0.01, real=1.14 secs]
>> 123.834: [CMS-concurrent-sweep-start]
>> 124.060: [GC 124.061: [ParNew
>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>> : 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 
>> 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>>  
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>   
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From jon.masamitsu at oracle.com  Tue Aug 31 16:14:36 2010
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Tue, 31 Aug 2010 16:14:36 -0700
Subject: iCMS question
In-Reply-To: <AANLkTinGk8S7hZ6OKq-B3Y+dqypWYJA+D+1quhbQaVXi@mail.gmail.com>
References: <AANLkTi=D23ym9Nj7mv68oucB1wKfcUrh=b64A8N4Oqyu@mail.gmail.com>
	<4C7D6C0B.9070103@oracle.com>
	<AANLkTinGk8S7hZ6OKq-B3Y+dqypWYJA+D+1quhbQaVXi@mail.gmail.com>
Message-ID: <4C7D8CDC.3040301@oracle.com>

If a larger heap helps, consider using the survivor spaces again.
Most of the time the survivor spaces help filter out short-lived
objects from the CMS gen and keeps fragmentation down.

Good luck.

On 08/31/10 14:05, Adam Hawthorne wrote:
> Thanks, somehow I missed the obvious answer.
>
> Adam
>
> --
> Adam Hawthorne
> Software Engineer
> BASIS International Ltd.
> www.basis.com <http://www.basis.com>
> +1.505.345.5232 Phone
>
>
> On Tue, Aug 31, 2010 at 16:54, Jon Masamitsu <jon.masamitsu at oracle.com 
> <mailto:jon.masamitsu at oracle.com>> wrote:
>
>     I think the short answer is that your heap is full.
>
>     This is a minor collection that turns into a full collection (so you
>     don't see the telltale "Full GC" message).
>
>
>     2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew
>     (promotion failed): 235520K->235520K(235712K), 1.0952192
>     secs]38711.901: [CMS2010-08-26T15:45:14.222+0200: 38711.903:
>     [CMS-concurrent-mark: 1.934/5.410 secs] [Times: user=8.20
>     sys=0.25, real=5.41 secs] 
>      (concurrent mode failure): 943043K->933853K(943744K), 4.0906398
>     secs] 1172473K->933853K(1179456K), [CMS Perm :
>     97361K->97361K(131072K)] icms_dc=86 , 5.1861184 secs] [Times:
>     user=5.18 sys=0.00, real=5.18 secs]
>
>     After the full collection the heap is still mostly full.
>
>     943043K->933853K(943744K)
>
>     I think your advice to increase the size of the should will help.
>
>
>     On 08/31/10 12:36, Adam Hawthorne wrote:
>>     Hi all,
>>
>>     I'm looking at an i-cms log, and I don't understand why I'm
>>     seeing what I'm seeing.  Any help is appreciated.  At about 
>>     38718.035 in the log below, icms_dc begins to report 100, and
>>     just after that I see several back-to-back concurrent mode
>>     failures.  During that time, the ParNew collections are reporting
>>     that they're doing no work.  This happens on and off for about 20
>>     minutes in the log.  Obviously the collections are taking
>>     seconds, but I don't see a Full GC every time.  What is it doing
>>     when it's not doing a full GC?  Is the continual CMS collection
>>     preventing the young generation from being collected (or perhaps
>>     collecting it simultaneously)?  When icms_dc=100, does that imply
>>     the application is stopped?
>>
>>     There was some uninformed command line twiddling going on here,
>>     so the GC parameters were:
>>     -Xmx1152m
>>     -Xms256m
>>     -XX:NewRatio=4
>>     -XX:MaxPermSize=128m
>>     -XX:+UseParNewGC
>>     -XX:-CMSParallelRemarkEnabled
>>     -XX:+UseConcMarkSweepGC
>>     -XX:+CMSPermGenSweepingEnabled
>>     -XX:+UseCMSCompactAtFullCollection
>>     -XX:CMSFullGCsBeforeCompaction=1
>>     -XX:+CMSClassUnloadingEnabled
>>     -XX:+CMSIncrementalMode
>>     -XX:+CMSIncrementalPacing
>>     -XX:CMSInitiatingOccupancyFraction=50
>>     -XX:ParallelGCThreads=1
>>     -XX:MaxTenuringThreshold=0
>>     -XX:SurvivorRatio=1024
>>     -XX:SoftRefLRUPolicyMSPerMB=1000
>>     -XX:+DisableExplicitGC
>>     -XX:+PrintGCDetails
>>     -XX:+PrintGCTimeStamps
>>     -XX:+PrintGCDateStamps
>>
>>     I had them
>>     remove -XX:ParallelGCThreads=1, -XX:MaxTenuringThreshold=0,
>>     and -XX:SurvivorRatio=1024 , thinking that since the duty cycle
>>     was approaching 100%, and because the ParNew pause times were so
>>     low, we could probably afford to spend more time doing young gc's
>>     instead of promoting.  I'm thinking I should probably tell them
>>     to increase their -Xmx and decrease their NewRatio as well.  Is
>>     there anything else?  I haven't done much tuning of i-cms in
>>     general, but the standard i-cms safety options don't seem like
>>     they would help here.
>>
>>     Here's the log excerpt:
>>
>>     2010-08-26T15:44:42.836+0200: 38680.510: [GC 38680.510: [ParNew:
>>     235520K->0K(235712K), 0.0187551 secs] 1158126K->923156K(1179456K)
>>     icms_dc=41 , 0.0189305 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:43.179+0200: 38680.861: [GC [1 CMS-initial-mark:
>>     923156K(943744K)] 996964K(1179456K), 0.0526219 secs] [Times:
>>     user=0.06 sys=0.00, real=0.06 secs] 
>>     2010-08-26T15:44:43.241+0200: 38680.914: [CMS-concurrent-mark-start]
>>     2010-08-26T15:44:44.021+0200: 38681.695: [GC 38681.695: [ParNew:
>>     235520K->0K(235712K), 0.0196054 secs] 1158676K->923705K(1179456K)
>>     icms_dc=41 , 0.0197785 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:45.160+0200: 38682.831: [GC 38682.831: [ParNew:
>>     235520K->0K(235712K), 0.0199663 secs] 1159225K->924310K(1179456K)
>>     icms_dc=41 , 0.0201448 secs] [Times: user=0.01 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:46.127+0200: 38683.797: [GC 38683.797: [ParNew:
>>     235520K->0K(235712K), 0.0549963 secs] 1159830K->930710K(1179456K)
>>     icms_dc=56 , 0.0551800 secs] [Times: user=0.05 sys=0.00,
>>     real=0.05 secs] 
>>     2010-08-26T15:44:47.562+0200: 38685.246: [CMS-concurrent-mark:
>>     1.903/4.332 secs] [Times: user=6.77 sys=0.25, real=4.32 secs] 
>>     2010-08-26T15:44:47.562+0200: 38685.246:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:44:47.828+0200: 38685.506: [GC 38685.507: [ParNew:
>>     235520K->0K(235712K), 0.0180149 secs] 1166230K->931150K(1179456K)
>>     icms_dc=56 , 0.0182051 secs] [Times: user=0.02 sys=0.00,
>>     real=0.01 secs] 
>>     2010-08-26T15:44:48.108+0200: 38685.790:
>>     [CMS-concurrent-preclean: 0.023/0.543 secs] [Times: user=0.55
>>     sys=0.05, real=0.55 secs] 
>>     2010-08-26T15:44:48.108+0200: 38685.790:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:44:48.483+0200: 38686.158:
>>     [CMS-concurrent-abortable-preclean: 0.040/0.368 secs] [Times:
>>     user=0.41 sys=0.02, real=0.38 secs] 
>>     2010-08-26T15:44:48.483+0200: 38686.159: [GC[YG occupancy: 136856
>>     K (235712 K)]38686.159: [Rescan (non-parallel) 38686.159: [grey
>>     object rescan, 0.0041031 secs]38686.163: [root rescan, 0.1153851
>>     secs], 0.1195876 secs]38686.279: [weak refs processing, 0.0012088
>>     secs]38686.280: [class unloading, 0.0143079 secs]38686.294:
>>     [scrub symbol & string tables, 0.0159790 secs] [1 CMS-remark:
>>     931150K(943744K)] 1068006K(1179456K), 0.1563910 secs] [Times:
>>     user=0.16 sys=0.00, real=0.16 secs] 
>>     2010-08-26T15:44:48.639+0200: 38686.316: [CMS-concurrent-sweep-start]
>>     2010-08-26T15:44:49.138+0200: 38686.808: [GC 38686.808: [ParNew:
>>     235520K->0K(235712K), 0.0183463 secs] 1166540K->931466K(1179456K)
>>     icms_dc=56 , 0.0185256 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-sweep:
>>     0.470/0.992 secs] [Times: user=1.47 sys=0.02, real=1.00 secs] 
>>     2010-08-26T15:44:49.637+0200: 38687.307: [CMS-concurrent-reset-start]
>>     2010-08-26T15:44:49.637+0200: 38687.313: [CMS-concurrent-reset:
>>     0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:44:50.277+0200: 38687.948: [GC 38687.948: [ParNew:
>>     235520K->0K(235712K), 0.0177987 secs] 1158207K->923110K(1179456K)
>>     icms_dc=42 , 0.0179840 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:50.604+0200: 38688.280: [GC [1 CMS-initial-mark:
>>     923110K(943744K)] 992484K(1179456K), 0.0509414 secs] [Times:
>>     user=0.05 sys=0.00, real=0.05 secs] 
>>     2010-08-26T15:44:50.651+0200: 38688.331: [CMS-concurrent-mark-start]
>>     2010-08-26T15:44:51.478+0200: 38689.157: [GC 38689.157: [ParNew:
>>     235520K->0K(235712K), 0.0178124 secs] 1158630K->923567K(1179456K)
>>     icms_dc=42 , 0.0179821 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:52.632+0200: 38690.306: [GC 38690.306: [ParNew:
>>     235520K->0K(235712K), 0.0180970 secs] 1159087K->924012K(1179456K)
>>     icms_dc=42 , 0.0182661 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:53.740+0200: 38691.422: [GC 38691.422: [ParNew:
>>     235520K->0K(235712K), 0.0267182 secs] 1159532K->925696K(1179456K)
>>     icms_dc=42 , 0.0269020 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:44:54.520+0200: 38692.201: [CMS-concurrent-mark:
>>     1.894/3.870 secs] [Times: user=5.74 sys=0.13, real=3.87 secs] 
>>     2010-08-26T15:44:54.520+0200: 38692.201:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:44:54.551+0200: 38692.222:
>>     [CMS-concurrent-preclean: 0.015/0.020 secs] [Times: user=0.05
>>     sys=0.02, real=0.03 secs] 
>>     2010-08-26T15:44:54.551+0200: 38692.222:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:44:54.894+0200: 38692.573: [GC 38692.573: [ParNew:
>>     235520K->0K(235712K), 0.0186037 secs] 1161216K->926149K(1179456K)
>>     icms_dc=42 , 0.0187843 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:55.581+0200: 38693.250:
>>     [CMS-concurrent-abortable-preclean: 0.050/1.028 secs] [Times:
>>     user=1.05 sys=0.01, real=1.03 secs] 
>>     2010-08-26T15:44:55.581+0200: 38693.251: [GC[YG occupancy: 140686
>>     K (235712 K)]38693.251: [Rescan (non-parallel) 38693.251: [grey
>>     object rescan, 0.0043953 secs]38693.255: [root rescan, 0.1209735
>>     secs], 0.1254680 secs]38693.376: [weak refs processing, 0.0013433
>>     secs]38693.378: [class unloading, 0.0142815 secs]38693.392:
>>     [scrub symbol & string tables, 0.0160091 secs] [1 CMS-remark:
>>     926149K(943744K)] 1066835K(1179456K), 0.1624119 secs] [Times:
>>     user=0.16 sys=0.00, real=0.16 secs] 
>>     2010-08-26T15:44:55.737+0200: 38693.414: [CMS-concurrent-sweep-start]
>>     2010-08-26T15:44:56.205+0200: 38693.879: [GC 38693.880: [ParNew:
>>     235520K->0K(235712K), 0.0188069 secs] 1161648K->926599K(1179456K)
>>     icms_dc=42 , 0.0189967 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-sweep:
>>     0.474/1.149 secs] [Times: user=1.64 sys=0.02, real=1.15 secs] 
>>     2010-08-26T15:44:56.891+0200: 38694.563: [CMS-concurrent-reset-start]
>>     2010-08-26T15:44:56.891+0200: 38694.569: [CMS-concurrent-reset:
>>     0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:44:57.343+0200: 38695.018: [GC 38695.018: [ParNew:
>>     235520K->0K(235712K), 0.0187471 secs] 1156567K->921554K(1179456K)
>>     icms_dc=32 , 0.0189332 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:44:57.718+0200: 38695.402: [GC [1 CMS-initial-mark:
>>     921554K(943744K)] 1001837K(1179456K), 0.0572313 secs] [Times:
>>     user=0.06 sys=0.00, real=0.06 secs] 
>>     2010-08-26T15:44:57.780+0200: 38695.460: [CMS-concurrent-mark-start]
>>     2010-08-26T15:44:58.545+0200: 38696.224: [GC 38696.225: [ParNew:
>>     235520K->0K(235712K), 0.0183404 secs] 1157074K->922009K(1179456K)
>>     icms_dc=32 , 0.0185097 secs] [Times: user=0.02 sys=0.00,
>>     real=0.01 secs] 
>>     2010-08-26T15:45:00.167+0200: 38697.850: [GC 38697.850: [ParNew:
>>     235520K->0K(235712K), 0.0196135 secs] 1157529K->922547K(1179456K)
>>     icms_dc=32 , 0.0197977 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:01.321+0200: 38698.992: [GC 38698.992: [ParNew:
>>     235520K->0K(235712K), 0.0187905 secs] 1158067K->923023K(1179456K)
>>     icms_dc=32 , 0.0189818 secs] [Times: user=0.01 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:02.476+0200: 38700.146: [GC 38700.146: [ParNew:
>>     235520K->0K(235712K), 0.0180207 secs] 1158543K->923471K(1179456K)
>>     icms_dc=32 , 0.0181892 secs] [Times: user=0.01 sys=0.00,
>>     real=0.01 secs] 
>>     2010-08-26T15:45:03.615+0200: 38701.284: [GC 38701.284: [ParNew:
>>     235520K->0K(235712K), 0.0186408 secs] 1158991K->923976K(1179456K)
>>     icms_dc=32 , 0.0188287 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:04.051+0200: 38701.735: [CMS-concurrent-mark:
>>     1.899/6.275 secs] [Times: user=8.16 sys=0.06, real=6.27 secs] 
>>     2010-08-26T15:45:04.051+0200: 38701.735:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:45:04.083+0200: 38701.756:
>>     [CMS-concurrent-preclean: 0.016/0.021 secs] [Times: user=0.05
>>     sys=0.02, real=0.03 secs] 
>>     2010-08-26T15:45:04.083+0200: 38701.756:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:45:04.753+0200: 38702.434: [GC 38702.435: [ParNew:
>>     235520K->0K(235712K), 0.0184004 secs] 1159496K->924427K(1179456K)
>>     icms_dc=32 , 0.0185851 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:05.409+0200: 38703.087:
>>     [CMS-concurrent-abortable-preclean: 0.063/1.331 secs] [Times:
>>     user=1.39 sys=0.03, real=1.33 secs] 
>>     2010-08-26T15:45:05.409+0200: 38703.088: [GC[YG occupancy: 141725
>>     K (235712 K)]38703.088: [Rescan (non-parallel) 38703.088: [grey
>>     object rescan, 0.0041240 secs]38703.093: [root rescan, 0.1200361
>>     secs], 0.1242579 secs]38703.213: [weak refs processing, 0.0010203
>>     secs]38703.214: [class unloading, 0.0142923 secs]38703.228:
>>     [scrub symbol & string tables, 0.0170438 secs] [1 CMS-remark:
>>     924427K(943744K)] 1066152K(1179456K), 0.1618864 secs] [Times:
>>     user=0.17 sys=0.00, real=0.17 secs] 
>>     2010-08-26T15:45:05.580+0200: 38703.250: [CMS-concurrent-sweep-start]
>>     2010-08-26T15:45:06.017+0200: 38703.687: [GC 38703.687: [ParNew:
>>     235520K->0K(235712K), 0.0425827 secs] 1159944K->929053K(1179456K)
>>     icms_dc=32 , 0.0427672 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:07.202+0200: 38704.873: [GC 38704.873: [ParNew:
>>     235520K->0K(235712K), 0.0179344 secs] 1161562K->926484K(1179456K)
>>     icms_dc=32 , 0.0181045 secs] [Times: user=0.01 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-sweep:
>>     0.472/2.032 secs] [Times: user=2.53 sys=0.08, real=2.03 secs] 
>>     2010-08-26T15:45:07.608+0200: 38705.282: [CMS-concurrent-reset-start]
>>     2010-08-26T15:45:07.608+0200: 38705.288: [CMS-concurrent-reset:
>>     0.005/0.005 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:08.341+0200: 38706.012: [GC 38706.012: [ParNew:
>>     235520K->0K(235712K), 0.0180667 secs] 1162004K->926949K(1179456K)
>>     icms_dc=24 , 0.0182417 secs] [Times: user=0.02 sys=0.00,
>>     real=0.02 secs] 
>>     2010-08-26T15:45:08.747+0200: 38706.425: [GC [1 CMS-initial-mark:
>>     926949K(943744K)] 1019208K(1179456K), 0.0677621 secs] [Times:
>>     user=0.06 sys=0.00, real=0.06 secs] 
>>     2010-08-26T15:45:08.809+0200: 38706.493: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:09.480+0200: 38707.158: [GC 38707.159: [ParNew:
>>     235520K->0K(235712K), 0.0477314 secs] 1162469K->932300K(1179456K)
>>     icms_dc=39 , 0.0479217 secs] [Times: user=0.05 sys=0.00,
>>     real=0.05 secs] 
>>     2010-08-26T15:45:10.463+0200: 38708.133: [GC 38708.133: [ParNew:
>>     235520K->0K(235712K), 0.0371965 secs] 1167820K->936491K(1179456K)
>>     icms_dc=54 , 0.0373840 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:12.085+0200: 38709.770: [GC 38709.770: [ParNew:
>>     235520K->0K(235712K), 0.0185757 secs] 1172011K->936953K(1179456K)
>>     icms_dc=69 , 0.0187606 secs] [Times: user=0.03 sys=0.00,
>>     real=0.03 secs] 
>>     2010-08-26T15:45:13.130+0200: 38710.806: [GC 38710.806: [ParNew
>>     (promotion failed): 235520K->235520K(235712K), 1.0952192
>>     secs]38711.901: [CMS2010-08-26T15:45:14.222+0200: 38711.903:
>>     [CMS-concurrent-mark: 1.934/5.410 secs] [Times: user=8.20
>>     sys=0.25, real=5.41 secs] 
>>      (concurrent mode failure): 943043K->933853K(943744K), 4.0906398
>>     secs] 1172473K->933853K(1179456K), [CMS Perm :
>>     97361K->97361K(131072K)] icms_dc=86 , 5.1861184 secs] [Times:
>>     user=5.18 sys=0.00, real=5.18 secs] 
>>     2010-08-26T15:45:18.372+0200: 38716.052: [GC [1 CMS-initial-mark:
>>     933853K(943744K)] 950390K(1179456K), 0.0155216 secs] [Times:
>>     user=0.01 sys=0.00, real=0.02 secs] 
>>     2010-08-26T15:45:18.387+0200: 38716.068: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:19.355+0200: 38717.035: [GC 38717.035: [ParNew:
>>     235520K->0K(235712K), 0.0623713 secs] 1169373K->942749K(1179456K)
>>     icms_dc=100 , 0.0625660 secs] [Times: user=0.06 sys=0.00,
>>     real=0.06 secs] 
>>     2010-08-26T15:45:20.509+0200: 38718.182: [CMS-concurrent-mark:
>>     1.964/2.114 secs] [Times: user=4.57 sys=0.11, real=2.12 secs] 
>>     2010-08-26T15:45:20.509+0200: 38718.182:
>>     [CMS-concurrent-preclean-start]
>>     2010-08-26T15:45:20.525+0200: 38718.202:
>>     [CMS-concurrent-preclean: 0.015/0.020 secs] [Times: user=0.02
>>     sys=0.02, real=0.02 secs] 
>>     2010-08-26T15:45:20.525+0200: 38718.202:
>>     [CMS-concurrent-abortable-preclean-start]
>>     2010-08-26T15:45:20.540+0200: 38718.222: [GC 38718.222: [ParNew:
>>     235520K->235520K(235712K), 0.0000592 secs]38718.222:
>>     [CMS2010-08-26T15:45:20.540+0200: 38718.223:
>>     [CMS-concurrent-abortable-preclean: 0.004/0.021 secs] [Times:
>>     user=0.02 sys=0.00, real=0.01 secs] 
>>      (concurrent mode failure): 942749K->936038K(943744K), 4.0650806
>>     secs] 1178269K->936038K(1179456K), [CMS Perm :
>>     97371K->97371K(131072K)] icms_dc=100 , 4.0653451 secs] [Times:
>>     user=4.07 sys=0.00, real=4.07 secs] 
>>     2010-08-26T15:45:24.612+0200: 38722.289: [GC [1 CMS-initial-mark:
>>     936038K(943744K)] 936042K(1179456K), 0.0023357 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:24.612+0200: 38722.291: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:26.141+0200: 38723.817: [GC 38723.817: [ParNew:
>>     235520K->235520K(235712K), 0.0000588 secs]38723.817:
>>     [CMS2010-08-26T15:45:27.030+0200: 38724.709:
>>     [CMS-concurrent-mark: 1.935/2.418 secs] [Times: user=3.65
>>     sys=0.19, real=2.42 secs] 
>>      (concurrent mode failure): 936038K->942227K(943744K), 4.9718511
>>     secs] 1171558K->942227K(1179456K), [CMS Perm :
>>     97393K->97393K(131072K)] icms_dc=100 , 4.9721115 secs] [Times:
>>     user=4.98 sys=0.00, real=4.98 secs] 
>>     2010-08-26T15:45:31.117+0200: 38728.790: [GC [1 CMS-initial-mark:
>>     942227K(943744K)] 942270K(1179456K), 0.0025839 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:31.117+0200: 38728.793: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:32.084+0200: 38729.758: [GC 38729.759: [ParNew:
>>     235520K->235520K(235712K), 0.0000623 secs]38729.759:
>>     [CMS2010-08-26T15:45:33.051+0200: 38730.736:
>>     [CMS-concurrent-mark: 1.934/1.943 secs] [Times: user=3.28
>>     sys=0.19, real=1.93 secs] 
>>      (concurrent mode failure): 942227K->943743K(943744K), 4.9442127
>>     secs] 1177747K->948024K(1179456K), [CMS Perm :
>>     97396K->97396K(131072K)] icms_dc=100 , 4.9446265 secs] [Times:
>>     user=4.91 sys=0.02, real=4.94 secs] 
>>     2010-08-26T15:45:37.029+0200: 38734.704: [GC [1 CMS-initial-mark:
>>     943743K(943744K)] 948026K(1179456K), 0.0067733 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:37.029+0200: 38734.711: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:38.745+0200: 38736.419: [Full GC 38736.419:
>>     [CMS2010-08-26T15:45:39.463+0200: 38737.138:
>>     [CMS-concurrent-mark: 1.952/2.427 secs] [Times: user=3.95
>>     sys=0.16, real=2.43 secs] 
>>      (concurrent mode failure): 943743K->943743K(943744K), 4.7856092
>>     secs] 1179455K->949021K(1179456K), [CMS Perm :
>>     97406K->97406K(131072K)] icms_dc=100 , 4.7857927 secs] [Times:
>>     user=4.79 sys=0.00, real=4.79 secs] 
>>     2010-08-26T15:45:43.534+0200: 38741.206: [GC [1 CMS-initial-mark:
>>     943743K(943744K)] 949287K(1179456K), 0.0080127 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:43.534+0200: 38741.214: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:44.455+0200: 38742.136: [Full GC 38742.136:
>>     [CMS2010-08-26T15:45:45.484+0200: 38743.168:
>>     [CMS-concurrent-mark: 1.952/1.954 secs] [Times: user=3.26
>>     sys=0.05, real=1.95 secs] 
>>      (concurrent mode failure): 943743K->943743K(943744K), 4.9692415
>>     secs] 1179440K->949759K(1179456K), [CMS Perm :
>>     97406K->97406K(131072K)] icms_dc=100 , 4.9694250 secs] [Times:
>>     user=4.96 sys=0.02, real=4.98 secs] 
>>     2010-08-26T15:45:49.883+0200: 38747.569: [GC [1 CMS-initial-mark:
>>     943743K(943744K)] 953372K(1179456K), 0.0086197 secs] [Times:
>>     user=0.02 sys=0.00, real=0.02 secs] 
>>     2010-08-26T15:45:49.899+0200: 38747.578: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:50.960+0200: 38748.634: [Full GC 38748.634:
>>     [CMS2010-08-26T15:45:51.833+0200: 38749.515:
>>     [CMS-concurrent-mark: 1.927/1.938 secs] [Times: user=3.23
>>     sys=0.11, real=1.93 secs] 
>>      (concurrent mode failure): 943743K->937800K(943744K), 4.9818453
>>     secs] 1179455K->937800K(1179456K), [CMS Perm :
>>     97407K->97407K(131072K)] icms_dc=100 , 4.9820296 secs] [Times:
>>     user=4.95 sys=0.02, real=4.98 secs] 
>>     2010-08-26T15:45:55.936+0200: 38753.617: [GC [1 CMS-initial-mark:
>>     937800K(943744K)] 938528K(1179456K), 0.0024408 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:45:55.936+0200: 38753.620: [CMS-concurrent-mark-start]
>>     2010-08-26T15:45:56.950+0200: 38754.630: [GC 38754.631: [ParNew:
>>     235520K->235520K(235712K), 0.0000597 secs]38754.631:
>>     [CMS2010-08-26T15:45:57.995+0200: 38755.679:
>>     [CMS-concurrent-mark: 2.057/2.059 secs] [Times: user=3.29
>>     sys=0.23, real=2.06 secs] 
>>      (concurrent mode failure): 937800K->936173K(943744K), 5.1274815
>>     secs] 1173320K->936173K(1179456K), [CMS Perm :
>>     97407K->97407K(131072K)] icms_dc=100 , 5.1277524 secs] [Times:
>>     user=5.13 sys=0.00, real=5.13 secs] 
>>     2010-08-26T15:46:02.535+0200: 38760.215: [GC [1 CMS-initial-mark:
>>     936173K(943744K)] 936184K(1179456K), 0.0024186 secs] [Times:
>>     user=0.00 sys=0.00, real=0.00 secs] 
>>     2010-08-26T15:46:02.535+0200: 38760.218: [CMS-concurrent-mark-start]
>>     2010-08-26T15:46:03.611+0200: 38761.290: [GC 38761.290: [ParNew
>>     (promotion failed): 235520K->235520K(235712K), 0.1302956
>>     secs]38761.420: [CMS2010-08-26T15:46:04.609+0200: 38762.289:
>>     [CMS-concurrent-mark: 1.938/2.071 secs] [Times: user=3.54
>>     sys=0.11, real=2.07 secs] 
>>      (concurrent mode failure): 943229K->939271K(943744K), 4.9629545
>>     secs] 1171693K->939271K(1179456K), [CMS Perm :
>>     97407K->97407K(131072K)] icms_dc=100 , 5.0934938 secs] [Times:
>>     user=5.09 sys=0.00, real=5.10 secs] 
>>
>>     Thanks again for any help,
>>
>>     Adam
>>
>>     --
>>     Adam Hawthorne
>>     Software Engineer
>>     BASIS International Ltd.
>>     www.basis.com <http://www.basis.com>
>>     +1.505.345.5232 Phone
>>     ------------------------------------------------------------------------
>>
>>     _______________________________________________
>>     hotspot-gc-use mailing list
>>     hotspot-gc-use at openjdk.java.net <mailto:hotspot-gc-use at openjdk.java.net>
>>     http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>       
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/attachments/20100831/1bc6f0cb/attachment-0001.html 
-------------- next part --------------
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

From y.s.ramakrishna at oracle.com  Tue Aug 31 16:57:36 2010
From: y.s.ramakrishna at oracle.com (Y. S. Ramakrishna)
Date: Tue, 31 Aug 2010 16:57:36 -0700
Subject: Long GC pause time
In-Reply-To: <4C7D81CD.5050706@oracle.com>
References: <SNT108-W11C7575D28055DFA3A6544E38A0@phx.gbl>	<4C7D49E4.6060908@oracle.com>
	<4C7D81CD.5050706@oracle.com>
Message-ID: <4C7D96F0.1060600@oracle.com>

oh, and -XX:+PrintReferenceGC to get more detail on
what kind of Reference objects are causing the issue.

-- ramki

On 08/31/10 15:27, Y. S. Ramakrishna wrote:
> Noam, also please provision and use the survivor spaces to age
> objects and reduce pressure on CMS. The premature
> promotion and subsequent death is probably exacerbating the
> proliferating Ref object deaths (if i may be permitted an
> oxymoron) in the old gen.
> 
> -- ramki
> 
> On 08/31/10 11:28, Jon Masamitsu wrote:
>> Try -XX+ParallelRefProcEnabled
>>
>> Your spending much of the pause time doing Reference processing
>>
>> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) , 
>> 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
>> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
>> [Times: user=1.68 sys=0.01, real=1.41 secs]
>>
>> (see the "[weak refs processing, 1.3860230 secs]") so doing it in 
>> parallel might
>> help.  ParallelRefProcEnabled isn't always a win so it's not "on" by 
>> default.
>>
>> You might also try -XX:+CMSScavengeBeforeRemark.  That's a just a guess.
>> Sometimes doing a minor collection before the remark helps.
>>
>> On 08/31/10 08:22, noam katzowicz wrote:
>>> Hello.
>>>  
>>> We are running JBoss AS over RH5, using java 6u21, with the following 
>>> GC params:
>>>  
>>> -Xms6750m -Xmx6750m
>>> -XX:+UseConcMarkSweepGC
>>> -XX:SurvivorRatio=128
>>> -XX:MaxTenuringThreshold=0
>>> -XX:+UseCompressedOops
>>> -XX:CMSInitiatingOccupancyFraction=60
>>> -Dsun.rmi.dgc.client.gcInterval=3600000
>>> -Dsun.rmi.dgc.server.gcInterval=3600000
>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution
>>>  
>>> We are experiencing relatively long GC pause time of over 1 second.
>>> Our application allocates hundreds of MBs per second. We?ve tried 
>>> different generation sizes but the above was the best we came up with. 
>>> Your help is appreciated.
>>>  
>>>
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0596370 secs] 4422502K->4173928K(6909888K), 
>>> 0.0606730 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>>> 89.486: [CMS-concurrent-preclean: 1.308/1.649 secs] [Times: user=10.32 
>>> sys=2.30, real=1.65 secs]
>>> 89.487: [CMS-concurrent-abortable-preclean-start]
>>> 89.804: [GC 89.804: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0603670 secs] 4446568K->4198190K(6909888K), 
>>> 0.0614310 secs] [Times: user=0.48 sys=0.03, real=0.07 secs]
>>> 89.866: [CMS-concurrent-abortable-preclean: 0.293/0.379 secs] [Times: 
>>> user=2.42 sys=0.54, real=0.38 secs]
>>> 89.867: [GC[YG occupancy: 420 K (274752 K)]89.867: [Rescan (parallel) 
>>> , 0.0242770 secs]89.892: [weak refs processing, 1.3860230 secs] [1 
>>> CMS-remark: 4198190K(6635136K)] 4198611K(6909888K), 1.4164670 secs] 
>>> [Times: user=1.68 sys=0.01, real=1.41 secs]
>>> 91.284: [CMS-concurrent-sweep-start]
>>> 91.590: [GC 91.591: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0675550 secs] 4371287K->4128966K(6909888K), 
>>> 0.0683470 secs] [Times: user=0.56 sys=0.01, real=0.07 secs]
>>> 91.860: [GC 91.860: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0604320 secs] 4308368K->4053829K(6909888K), 
>>> 0.0612020 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>>> 92.786: [GC 92.787: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.1028730 secs] 3964957K->3753351K(6909888K), 
>>> 0.1038490 secs] [Times: user=0.80 sys=0.02, real=0.10 secs]
>>> 93.106: [GC 93.107: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0487000 secs] 3949085K->3699496K(6909888K), 
>>> 0.0494470 secs] [Times: user=0.32 sys=0.01, real=0.05 secs]
>>> 93.371: [GC 93.371: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0430620 secs] 3905862K->3652769K(6909888K), 
>>> 0.0438800 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
>>> 94.637: [GC 94.638: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0564930 secs] 3416325K->3164258K(6909888K), 
>>> 0.0575000 secs] [Times: user=0.44 sys=0.02, real=0.06 secs]
>>> 94.863: [CMS-concurrent-sweep: 3.107/3.578 secs] [Times: user=16.85 
>>> sys=3.23, real=3.58 secs]
>>> 94.863: [CMS-concurrent-reset-start]
>>> 94.878: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 
>>> sys=0.01, real=0.02 secs]
>>> 95.981: [GC 95.981: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0514980 secs] 3405940K->3152334K(6909888K), 
>>> 0.0524610 secs] [Times: user=0.44 sys=0.01, real=0.05 secs]
>>> 100.179: [GC 100.180: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0676510 secs] 3424974K->3178150K(6909888K), 
>>> 0.0686240 secs] [Times: user=0.58 sys=0.00, real=0.07 secs]
>>> 100.909: [GC 100.909: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0707060 secs] 3450790K->3202638K(6909888K), 
>>> 0.0717030 secs] [Times: user=0.55 sys=0.01, real=0.07 secs]
>>> 101.613: [GC 101.613: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272636K->0K(274752K), 0.0667270 secs] 3475274K->3224061K(6909888K), 
>>> 0.0678240 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>>> 102.072: [GC 102.073: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272635K->0K(274752K), 0.0590350 secs] 3496696K->3241246K(6909888K), 
>>> 0.0601190 secs] [Times: user=0.44 sys=0.00, real=0.06 secs]
>>> 102.756: [GC 102.756: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272613K->0K(274752K), 0.0630060 secs] 3513860K->3262602K(6909888K), 
>>> 0.0640320 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>>> 103.452: [GC 103.453: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0533390 secs] 3535242K->3280184K(6909888K), 
>>> 0.0543760 secs] [Times: user=0.41 sys=0.01, real=0.06 secs]
>>> 104.462: [GC 104.462: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0729890 secs] 3552824K->3299003K(6909888K), 
>>> 0.0740260 secs] [Times: user=0.48 sys=0.01, real=0.07 secs]
>>> 106.274: [GC 106.274: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0609880 secs] 3571643K->3314370K(6909888K), 
>>> 0.0619510 secs] [Times: user=0.41 sys=0.00, real=0.06 secs]
>>> 108.154: [GC 108.155: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.1251050 secs] 3587010K->3343996K(6909888K), 
>>> 0.1261630 secs] [Times: user=0.71 sys=0.02, real=0.13 secs]
>>> 108.882: [GC 108.883: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0952060 secs] 3616636K->3369490K(6909888K), 
>>> 0.0962930 secs] [Times: user=0.64 sys=0.01, real=0.10 secs]
>>> 109.482: [GC 109.482: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272936K->0K(274752K), 0.0791270 secs] 3642426K->3392040K(6909888K), 
>>> 0.0800830 secs] [Times: user=0.53 sys=0.01, real=0.08 secs]
>>> 109.948: [GC 109.949: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 273038K->0K(274752K), 0.0835410 secs] 3665079K->3417813K(6909888K), 
>>> 0.0845680 secs] [Times: user=0.57 sys=0.00, real=0.08 secs]
>>> 110.035: [GC 110.035: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 64K->0K(274752K), 0.0088490 secs] 3417877K->3417828K(6909888K), 
>>> 0.0093860 secs] [Times: user=0.09 sys=0.00, real=0.01 secs]
>>> 110.484: [GC 110.485: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0880810 secs] 3690468K->3443727K(6909888K), 
>>> 0.0890560 secs] [Times: user=0.59 sys=0.00, real=0.09 secs]
>>> 110.910: [GC 110.910: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0860790 secs] 3716367K->3475451K(6909888K), 
>>> 0.0870070 secs] [Times: user=0.58 sys=0.02, real=0.08 secs]
>>> 111.276: [GC 111.277: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272639K->0K(274752K), 0.0772230 secs] 3748091K->3503686K(6909888K), 
>>> 0.0782780 secs] [Times: user=0.47 sys=0.02, real=0.08 secs]
>>> 111.658: [GC 111.659: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0754700 secs] 3776326K->3532682K(6909888K), 
>>> 0.0763440 secs] [Times: user=0.54 sys=0.01, real=0.07 secs]
>>> 112.068: [GC 112.068: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0705680 secs] 3805322K->3559619K(6909888K), 
>>> 0.0714610 secs] [Times: user=0.47 sys=0.02, real=0.07 secs]
>>> 112.497: [GC 112.498: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0779750 secs] 3832259K->3589461K(6909888K), 
>>> 0.0788830 secs] [Times: user=0.50 sys=0.01, real=0.08 secs]
>>> 112.911: [GC 112.911: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0728900 secs] 3862101K->3616744K(6909888K), 
>>> 0.0738560 secs] [Times: user=0.52 sys=0.00, real=0.07 secs]
>>> 113.362: [GC 113.363: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0609030 secs] 3889384K->3638477K(6909888K), 
>>> 0.0619050 secs] [Times: user=0.44 sys=0.01, real=0.06 secs]
>>> 113.829: [GC 113.829: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272614K->0K(274752K), 0.0641940 secs] 3911091K->3659305K(6909888K), 
>>> 0.0653490 secs] [Times: user=0.48 sys=0.01, real=0.06 secs]
>>> 114.311: [GC 114.311: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0618170 secs] 3931945K->3681619K(6909888K), 
>>> 0.0626970 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>>> 114.841: [GC 114.842: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0670890 secs] 3954259K->3711036K(6909888K), 
>>> 0.0680540 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>>> 115.356: [GC 115.357: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0623700 secs] 3983676K->3737945K(6909888K), 
>>> 0.0633630 secs] [Times: user=0.49 sys=0.00, real=0.07 secs]
>>> 115.829: [GC 115.830: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0665900 secs] 4010585K->3766876K(6909888K), 
>>> 0.0676240 secs] [Times: user=0.51 sys=0.01, real=0.06 secs]
>>> 116.271: [GC 116.271: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272630K->0K(274752K), 0.0594280 secs] 4039506K->3796861K(6909888K), 
>>> 0.0603620 secs] [Times: user=0.46 sys=0.02, real=0.06 secs]
>>> 116.716: [GC 116.717: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0724150 secs] 4069501K->3825191K(6909888K), 
>>> 0.0733430 secs] [Times: user=0.52 sys=0.01, real=0.08 secs]
>>> 117.145: [GC 117.146: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0664990 secs] 4097831K->3853057K(6909888K), 
>>> 0.0675760 secs] [Times: user=0.48 sys=0.00, real=0.07 secs]
>>> 117.543: [GC 117.545: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0650350 secs] 4125697K->3880562K(6909888K), 
>>> 0.0673450 secs] [Times: user=0.49 sys=0.01, real=0.07 secs]
>>> 117.956: [GC 117.956: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0664410 secs] 4153202K->3907935K(6909888K), 
>>> 0.0673980 secs] [Times: user=0.49 sys=0.02, real=0.07 secs]
>>> 118.385: [GC 118.386: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272631K->0K(274752K), 0.0655070 secs] 4180566K->3934925K(6909888K), 
>>> 0.0664120 secs] [Times: user=0.50 sys=0.01, real=0.07 secs]
>>> 118.826: [GC 118.827: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272608K->0K(274752K), 0.0605390 secs] 4207533K->3964211K(6909888K), 
>>> 0.0616050 secs] [Times: user=0.50 sys=0.02, real=0.06 secs]
>>> 119.233: [GC 119.234: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0656870 secs] 4236851K->3991065K(6909888K), 
>>> 0.0666900 secs] [Times: user=0.50 sys=0.02, real=0.07 secs]
>>> 119.301: [GC [1 CMS-initial-mark: 3991065K(6635136K)] 
>>> 3991071K(6909888K), 0.0091650 secs] [Times: user=0.01 sys=0.00, 
>>> real=0.01 secs]
>>> 119.311: [CMS-concurrent-mark-start]
>>> 119.699: [GC 119.699: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0647090 secs] 4263705K->4019961K(6909888K), 
>>> 0.0656290 secs] [Times: user=0.53 sys=0.01, real=0.06 secs]
>>> 120.081: [GC 120.082: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0641110 secs] 4292601K->4048334K(6909888K), 
>>> 0.0649070 secs] [Times: user=0.50 sys=0.01, real=0.06 secs]
>>> 120.538: [GC 120.539: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0677200 secs] 4320974K->4075491K(6909888K), 
>>> 0.0687990 secs] [Times: user=0.49 sys=0.03, real=0.07 secs]
>>> 120.621: [CMS-concurrent-mark: 1.076/1.310 secs] [Times: user=10.66 
>>> sys=1.73, real=1.31 secs]
>>> 120.621: [CMS-concurrent-preclean-start]
>>> 120.942: [GC 120.943: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0736770 secs] 4348131K->4104969K(6909888K), 
>>> 0.0747430 secs] [Times: user=0.51 sys=0.03, real=0.07 secs]
>>> 121.337: [GC 121.338: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0569980 secs] 4377609K->4126623K(6909888K), 
>>> 0.0579070 secs] [Times: user=0.48 sys=0.03, real=0.06 secs]
>>> 121.766: [GC 121.767: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272615K->0K(274752K), 0.0590140 secs] 4399238K->4148770K(6909888K), 
>>> 0.0599510 secs] [Times: user=0.44 sys=0.03, real=0.06 secs]
>>> 122.174: [GC 122.175: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0531420 secs] 4421410K->4167895K(6909888K), 
>>> 0.0540310 secs] [Times: user=0.45 sys=0.03, real=0.05 secs]
>>> 122.257: [CMS-concurrent-preclean: 1.306/1.636 secs] [Times: 
>>> user=10.25 sys=2.21, real=1.63 secs]
>>> 122.257: [CMS-concurrent-abortable-preclean-start]
>>> 122.631: [GC 122.631: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0528220 secs] 4440535K->4187654K(6909888K), 
>>> 0.0537160 secs] [Times: user=0.46 sys=0.02, real=0.05 secs]
>>> 122.696: [CMS-concurrent-abortable-preclean: 0.355/0.439 secs] [Times: 
>>> user=2.50 sys=0.52, real=0.44 secs]
>>> 122.698: [GC[YG occupancy: 7874 K (274752 K)]122.698: [Rescan 
>>> (parallel) , 0.0253430 secs]122.723: [weak refs processing, 1.1038100 
>>> secs] [1 CMS-remark: 4187654K(6635136K)] 4195529K(6909888K), 1.1357970 
>>> secs] [Times: user=1.42 sys=0.01, real=1.14 secs]
>>> 123.834: [CMS-concurrent-sweep-start]
>>> 124.060: [GC 124.061: [ParNew
>>> Desired survivor size 1081344 bytes, new threshold 0 (max 0)
>>> : 272640K->0K(274752K), 0.0690300 secs] 4399395K->4167257K(6909888K), 
>>> 0.0698010 secs] [Times: user=0.52 sys=0.02, real=0.07 secs]
>>>  
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>   
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use