JDK 9 RFR of JDK-8078334: Mark regression tests using randomness
joe.darcy at oracle.com
Fri Apr 24 18:04:24 UTC 2015
On 4/23/2015 10:58 PM, Alan Bateman wrote:
> On 24/04/2015 02:54, Joseph D. Darcy wrote:
>> Any additional comments on marking with tests in question with a
>> "randomness" keyword?
> I don't object to this keyword although I should say that most of the
> (apparent) randomness that I've seen hasn't been because of tests
> using Random but rather because of non-deterministic sequence of tests
> running in the same VM. The switch from othervm to agentvm a few years
> ago took a long time to weed out issues, same thing (even more) with
> -concurrency. Another source of apparent randomness is just having a
> large pool of machines and where machines are randomly selected to
> execute the tests.
> I agree with your comments that tests using Random should have report
> the seed and a have a way to re-run with the same value. It could help
> with some cases.
> One thing that isn't clear to me is how this keyword will be
> maintained. If I modify a test and replace the use of Random then I
> guess I need to remember to also remove this label. When I add a test
> that uses Random ... Just wondering if there is a tool to do the
> tests analysis, identify the use of specific APIs ,and help identify
> where the keyword might be missing or not needed.
The goal of the keyword is to enable better analysis of bug failures. We
(still) have some intermittent test failures in the platform, some
probably remaining from -concurrency issues and some due to inconsistent
machine config issues on the test farms. However, there is reason to
believe some of the tests also fail because of the use of randomness in
the generation of test cases. For example, the test
fails about once every three hundred runs when the random cases it
generates trigger an error condition. Setting the seed to known-bad
value causes the test to fail consistently (JDK-8055269). I have
suspicions some of our other intermittently failing tests are in a
similar but undiagnosed situation.
If a test uses randomness, I think the default assumed cause for a test
failure should switch from environmental causes (bad machine config,
etc.) to the random behavior. Therefore, when a random-using test fails,
I think the failure analysis needs to include recording of the bad seed
to see if the failure is reproducible / deterministic with a fixed seed
I was very surprised by the large number of random-using tests in the
JDK regression tests. Nearly all of these have been stably passing for
many years. After the one-time cost of this large update to add the
keywords, there would be a small amount of incremental test maintenance
as new random-using tests were added or the randomness value of a test
changed. However, I would expect whether or not a test used randomness
to be a property of a test that very rarely changed.
More information about the security-dev