Test policy follow-up, third testing tier
joe.darcy at oracle.com
Wed Jul 1 17:56:26 UTC 2015
On 6/24/2015 3:22 AM, Martijn Verburg wrote:
> Hi Joe,
> Just a thought from the outside looking in (feel free to discard!).
> I'm seeing a reasonably strong trend to label these types of test
> suites/tiers as: Unit, Integration, System and Heisen. It's a very
> small thing, but perhaps OpenJDK as a project could use some of this
> (hopefully familiar) terminology. e.g. I know that if I was a new
> developer joining OpenJDK that if Tier 2 was re-labelled Heisen then
> I'd immediately know what that meant.
Thanks for the feedback on naming. The tiers as they are currently
constituted don't align well with the divisions you're outlined.
However, I think more informative naming for the tiers is something to
keep in mind to revisit in the future.
> Apart from that, I really like the proposal, I know it's been
> personally frustrating as a casual contributor to see a Heisen test
> fail when making a fairly trivial change.
> As an aside the Adoption Group has an unofficial nightly build of
> jtreg and some other code tools at:
> On 23 June 2015 at 23:48, joe darcy <joe.darcy at oracle.com
> <mailto:joe.darcy at oracle.com>> wrote:
> As a reminder, JDK 9 is now using a tiered testing policy where
> about 9,000 tests have been placed into one of two tiers, with
> tier 1 tests having a stricter policy on passing than tier 2. 
> The overall policy is that tier 1 tests should always pass in
> master and that integrations, both dev -> master and hotspot main
> -> dev, should preserve the property that all tier 1 tests pass.
> In addition, only a limited number of tier 2 tests should fail.
> In related work, jtreg keywords are now used to mark tests as
> using randomness and/or being known to fail intermittently. A test
> using randomness which is also known to fail intermittently should
> be modified so that the random seed is recorded in the test log
> and so that the seed can be set to see if the test reproducibly
> fails with a particular seed value.  Analysis of a test failure
> of a test using randomness should include recording the seed value
> in a bug.
> There are a number of expectations for regression tests in the
> JDK. There is a general expectation that a test passes reliably
> and quickly across platforms. Tests specific to particular
> platform can indicate that using the jtreg @requires feature. 
> Tests must pass with assertions and system assertions enabled, as
> controlled by the jtreg options -ea and -esa, respectively. In
> addition, tests should pass when invoked under jtreg's agent vm
> mode with concurrency, jtreg options -agentvm and -conc:$NUMBER.
> If a test for technical reasons cannot be run in agentvm mode, it
> should be written to explicitly run in othervm mode (@run
> main/othervm TestName) or use other equivalent measures, such as
> including the directory hosting the test on the othervm.dirs
> property in TEST.ROOT. That way such a test will still pass when
> invoked by a jtreg command which otherwise uses agentvm mode.
> Developers should use a current version of jtreg for test runs as
> the test suite is relying on features in new jtreg versions. 
> The jdk repo is declared to require at least jtreg 4.1 b11; jtreg
> 4.1 b12 is the latest version as of writing this message. One or
> more additional jtreg version can be expected before JDK 9 ships.
> The next iteration of refining tiered testing is to add a third
> tier, tier 3. The initial tier 1 and tier 2 test definitions did
> not include any client library tests. As mentioned as a
> possibility in the earlier tiered testing proposal, tier 3 can
> start hosting client library tests, beginning with sets of tests
> which do not require a "headful" environment.  Detailed
> discussions about which test sets to include will be conducted
> with the appropriate client libs teams. Additionally, for areas
> with known intermittent test problems, tests can be moved down to
> a lower test tier to ease getting clean results on the higher
> tier. For example, despite many improvements made over the last
> several years to the speed and robustness of the rmi tests, the
> rmi tests are still prone to intermittent failures when run
> concurrently. Therefore, as follow-up work after tier 3 is added,
> I propose moving the rmi tests from tier 2 to tier 3. 
>  http://openjdk.java.net/jtreg/tag-spec.html#requires_names
>  For download information, see http://openjdk.java.net/jtreg/
>  8081547: Prepare client libs regression tests for running in a
> concurrent, headless jtreg environment
>  JDK-8129624: Move jdk_rmi test group from tier 2 to tier 3
More information about the jdk9-dev