JDK 9 RFR of JDK-8078334: Mark regression tests using randomness

Joseph D. Darcy joe.darcy at oracle.com
Wed Apr 22 02:13:16 UTC 2015


As the next step of implementing "Proposed new policies for JDK 9 
regression tests: tiered testing, intermittent failures, and randomness" 
[1], please review this changeset which adds a "randomness" jtreg 
keyword and tags 240 regression tests in the jdk repository with the 

     JDK-8078334: Mark regression tests using randomness

The update to test/TEST.ROOT is

-# The list of keywords supported in the entire test suite
-keys=2d dnd i18n intermittent
+# The list of keywords supported in the entire test suite.  The
+# "intermittent" keyword marks tests known to fail intermittently.
+# The "randomness" keyword marks tests using randomness with test
+# cases differing from run to run. (A test using a fixed random seed
+# would not count as "randomness" by this definition.) Extra care
+# should be taken to handle test failures of intermittent or
+# randomness tests.
+keys=2d dnd i18n intermittent randomness

To find the affected files for this fix, I grepped over the regression 
tests looking for "random" and verified if a random number generated was 
used in a way where its values would differ from run to run. In 
particular, as noted above, a java.util.Random random number generator 
with a fixed seed did *not* qualify a test for the "randomness" keyword.

The usual update to the files in this changeset is just

+ * @key randomness

The updated tests are mostly in the core libs and security libs areas 
with a few serviceability tests too. The list in this bug is not quite 
exhaustive; I did not include rmi tests or client libraries tests.

I verified that running

     jtreg ... -l -k:randomness .

returns the expected number of tests and that the tests continue to pass 
(or be ignored) when run.

One goal of marking the tests using randomness is to help root out some 
remaining intermittent test failures. If one of the randomness tests is 
observed to fail intermittently, if it has not already been updated to 
print out the random seed and be able to accept a particular seed when 
run, the test should be so modified so that future failures can log the 
seed value in hopes of reproducing the failure.

Brian Burkhalter has proactively made these sorts of changes to the 
java.math tests earlier in JDK 9. [2] Since most of the randomness tests 
don't have a history of instability (and due to the unexpected large 
number of them!), I don't think it is necessary to proactively update 
all the other tests without evidence of a history of failure. However, 
when an otherwise unexplained failure is seen, I recommend applying the 
sort of updates Brian has done in java.math to the failing test(s).



[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html

[2] JDK-8074460, Always print seeds used in [Splittable]Random instances 
in java.math tests,

More information about the security-dev mailing list