JDK 9 RFR of JDK-8078334: Mark regression tests using randomness
Joseph D. Darcy
joe.darcy at oracle.com
Fri Apr 24 01:54:17 UTC 2015
Any additional comments on marking with tests in question with a
On 4/21/2015 7:13 PM, Joseph D. Darcy wrote:
> As the next step of implementing "Proposed new policies for JDK 9
> regression tests: tiered testing, intermittent failures, and
> randomness" , please review this changeset which adds a
> "randomness" jtreg keyword and tags 240 regression tests in the jdk
> repository with the keyword:
> JDK-8078334: Mark regression tests using randomness
> The update to test/TEST.ROOT is
> -# The list of keywords supported in the entire test suite
> -keys=2d dnd i18n intermittent
> +# The list of keywords supported in the entire test suite. The
> +# "intermittent" keyword marks tests known to fail intermittently.
> +# The "randomness" keyword marks tests using randomness with test
> +# cases differing from run to run. (A test using a fixed random seed
> +# would not count as "randomness" by this definition.) Extra care
> +# should be taken to handle test failures of intermittent or
> +# randomness tests.
> +keys=2d dnd i18n intermittent randomness
> To find the affected files for this fix, I grepped over the regression
> tests looking for "random" and verified if a random number generated
> was used in a way where its values would differ from run to run. In
> particular, as noted above, a java.util.Random random number generator
> with a fixed seed did *not* qualify a test for the "randomness" keyword.
> The usual update to the files in this changeset is just
> + * @key randomness
> The updated tests are mostly in the core libs and security libs areas
> with a few serviceability tests too. The list in this bug is not quite
> exhaustive; I did not include rmi tests or client libraries tests.
> I verified that running
> jtreg ... -l -k:randomness .
> returns the expected number of tests and that the tests continue to
> pass (or be ignored) when run.
> One goal of marking the tests using randomness is to help root out
> some remaining intermittent test failures. If one of the randomness
> tests is observed to fail intermittently, if it has not already been
> updated to print out the random seed and be able to accept a
> particular seed when run, the test should be so modified so that
> future failures can log the seed value in hopes of reproducing the
> Brian Burkhalter has proactively made these sorts of changes to the
> java.math tests earlier in JDK 9.  Since most of the randomness
> tests don't have a history of instability (and due to the unexpected
> large number of them!), I don't think it is necessary to proactively
> update all the other tests without evidence of a history of failure.
> However, when an otherwise unexplained failure is seen, I recommend
> applying the sort of updates Brian has done in java.math to the
> failing test(s).
>  JDK-8074460, Always print seeds used in [Splittable]Random
> instances in java.math tests,
More information about the security-dev