RFR: JDK-8176084 Developer-friendly run-test facility
jonathan.gibbons at oracle.com
Thu Mar 2 21:40:39 UTC 2017
I don't know if it helps or not, but jtreg defines the following exit codes:
1: No tests to run ... none specified, or no tests to run in the
2: Some tests failed ... jtreg ran the test and but the test did not pass
3: Some tests had an error ... jtreg could not run the test
4: Bad command-line args ... user error
5: Something else bad happened ... typically a configuration issue
6: Exception/crash ... oops
The intent is that they are ordered by seriousness, so that you can set
a simple numeric threshhold of what you consider an acceptable outcome.
On 3/2/17 1:25 PM, Magnus Ihse Bursie wrote:
> On 2017-03-02 15:37, Erik Joelsson wrote:
>> On 2017-03-02 14:48, Magnus Ihse Bursie wrote:
>>> On 2017-03-02 12:19, Erik Joelsson wrote:
>>>> I don't think I like this part. It's not uncommon to expect non
>>>> zero return when tests are failing even in developer sessions. If
>>>> we are to ever convert to using this new run-test for automated
>>>> systems, which we really should, it must return non zero on failures.
>>> While this is probably true, that's not the only thing that needs
>>> adapting for having this run in automated systems. At this point in
>>> time, the goal was limited to providing a good developer experience.
>>> I hope too that we can expand this framework for using it in
>>> distributed test systems, but that needs much more work, and will
>>> likely be more intrusive.
>>>> I'm guessing you added this to avoid the extra failure printing
>>>> from the build system.
>>> Well yes and no. In the old test framework, the behavior was not
>>> consistent whether to exit on failed tests result or not. I chose
>>> the stance that having successful make execution of the tests, even
>>> if some tests fails, warranted a successful make execution.
>> In which cases were there inconsistency? JPRT certainly relies on
>> make failing for everything it was running. That would at least cover
>> the vast majority of cases actually in use. I would go as far as
>> saying any other cases not currently conforming should be viewed as
> For instance, if you set TREAT_EXIT_CODE_1_AS_0 then, as the variable
> says, exit code 1 will be treated as exit code 0. For the langtools
> test Makefile, you could either set EXIT=true to avoid (!) having it
> exit on failure. Also, langtools jtreg uses a EXIT_IF_FATAL check
> which compares > FATAL_JTREG_EXIT = 3. In one of our internal test
> suites, the default behavior is to use exit code 0 even on test
> failures, and our current make wrapper looks for the existence of a
> file to trigger a non-zero exit. So it's a bit messy.
>>> This can of course be changed to the reverse so failed tests always
>>> lead to a make failure exit, or have the behavior selected by the user.
>>>> Surely this can be worked around differently?
>>> Yes, but not so unobtrusively. I wanted to have this change make a
>>> minimal impact on existing code.
>>> My suggestion is that we keep the current functionality, and work on
>>> getting a way to return non-zero results from test failures as part
>>> of a further development of this framework for distributed testing.
>> I suggest that we change it to fail on failed tests instead of
>> changing the behavior of the current test mechanism.
> I'm not sure I understand what you mean. My patch does not change any
> behavior of the current test mechanism. I will not introduce any such
> changes, not even if you wanted me to. :-) So that sounds like a false
> I can change the new run-test system to fail on failed test. To keep
> it as readable as it is now, I would need to make a hack with the
> top-level FailureHandler. This sure can be done, but increases the
> risk of unintended consequences. But since you feel so strongly about
> this, I assume that's the way to go.
>> There will likely be other changes needed before automated systems
>> can use this, but this is a very fundamental part of the API. If make
>> doesn't fail, any wrapping tool/script/system is unable to know if
>> the run was successful. All other build systems I know of that run
>> tests do this (gradle, maven, ant etc).
More information about the build-dev