Test policy follow-up, third testing tier
joe.darcy at oracle.com
Tue Jun 23 22:48:47 UTC 2015
As a reminder, JDK 9 is now using a tiered testing policy where about
9,000 tests have been placed into one of two tiers, with tier 1 tests
having a stricter policy on passing than tier 2.  The overall policy
is that tier 1 tests should always pass in master and that integrations,
both dev -> master and hotspot main -> dev, should preserve the property
that all tier 1 tests pass. In addition, only a limited number of tier 2
tests should fail.
In related work, jtreg keywords are now used to mark tests as using
randomness and/or being known to fail intermittently. A test using
randomness which is also known to fail intermittently should be modified
so that the random seed is recorded in the test log and so that the seed
can be set to see if the test reproducibly fails with a particular seed
value.  Analysis of a test failure of a test using randomness should
include recording the seed value in a bug.
There are a number of expectations for regression tests in the JDK.
There is a general expectation that a test passes reliably and quickly
across platforms. Tests specific to particular platform can indicate
that using the jtreg @requires feature.  Tests must pass with
assertions and system assertions enabled, as controlled by the jtreg
options -ea and -esa, respectively. In addition, tests should pass when
invoked under jtreg's agent vm mode with concurrency, jtreg options
-agentvm and -conc:$NUMBER. If a test for technical reasons cannot be
run in agentvm mode, it should be written to explicitly run in othervm
mode (@run main/othervm TestName) or use other equivalent measures, such
as including the directory hosting the test on the othervm.dirs property
in TEST.ROOT. That way such a test will still pass when invoked by a
jtreg command which otherwise uses agentvm mode.
Developers should use a current version of jtreg for test runs as the
test suite is relying on features in new jtreg versions.  The jdk
repo is declared to require at least jtreg 4.1 b11; jtreg 4.1 b12 is the
latest version as of writing this message. One or more additional jtreg
version can be expected before JDK 9 ships.
The next iteration of refining tiered testing is to add a third tier,
tier 3. The initial tier 1 and tier 2 test definitions did not include
any client library tests. As mentioned as a possibility in the earlier
tiered testing proposal, tier 3 can start hosting client library tests,
beginning with sets of tests which do not require a "headful"
environment.  Detailed discussions about which test sets to include
will be conducted with the appropriate client libs teams. Additionally,
for areas with known intermittent test problems, tests can be moved down
to a lower test tier to ease getting clean results on the higher tier.
For example, despite many improvements made over the last several years
to the speed and robustness of the rmi tests, the rmi tests are still
prone to intermittent failures when run concurrently. Therefore, as
follow-up work after tier 3 is added, I propose moving the rmi tests
from tier 2 to tier 3. 
 For download information, see http://openjdk.java.net/jtreg/
 8081547: Prepare client libs regression tests for running in a
concurrent, headless jtreg environment
 JDK-8129624: Move jdk_rmi test group from tier 2 to tier 3
More information about the jdk9-dev