RFR: JDK-8176084 Developer-friendly run-test facility
Magnus Ihse Bursie
magnus.ihse.bursie at oracle.com
Thu Mar 2 21:25:49 UTC 2017
On 2017-03-02 15:37, Erik Joelsson wrote:
> On 2017-03-02 14:48, Magnus Ihse Bursie wrote:
>> On 2017-03-02 12:19, Erik Joelsson wrote:
>>> I don't think I like this part. It's not uncommon to expect non zero
>>> return when tests are failing even in developer sessions. If we are
>>> to ever convert to using this new run-test for automated systems,
>>> which we really should, it must return non zero on failures.
>> While this is probably true, that's not the only thing that needs
>> adapting for having this run in automated systems. At this point in
>> time, the goal was limited to providing a good developer experience.
>> I hope too that we can expand this framework for using it in
>> distributed test systems, but that needs much more work, and will
>> likely be more intrusive.
>>> I'm guessing you added this to avoid the extra failure printing from
>>> the build system.
>> Well yes and no. In the old test framework, the behavior was not
>> consistent whether to exit on failed tests result or not. I chose the
>> stance that having successful make execution of the tests, even if
>> some tests fails, warranted a successful make execution.
> In which cases were there inconsistency? JPRT certainly relies on make
> failing for everything it was running. That would at least cover the
> vast majority of cases actually in use. I would go as far as saying
> any other cases not currently conforming should be viewed as bugs.
For instance, if you set TREAT_EXIT_CODE_1_AS_0 then, as the variable
says, exit code 1 will be treated as exit code 0. For the langtools test
Makefile, you could either set EXIT=true to avoid (!) having it exit on
failure. Also, langtools jtreg uses a EXIT_IF_FATAL check which compares
> FATAL_JTREG_EXIT = 3. In one of our internal test suites, the default
behavior is to use exit code 0 even on test failures, and our current
make wrapper looks for the existence of a file to trigger a non-zero
exit. So it's a bit messy.
>> This can of course be changed to the reverse so failed tests always
>> lead to a make failure exit, or have the behavior selected by the user.
>>> Surely this can be worked around differently?
>> Yes, but not so unobtrusively. I wanted to have this change make a
>> minimal impact on existing code.
>> My suggestion is that we keep the current functionality, and work on
>> getting a way to return non-zero results from test failures as part
>> of a further development of this framework for distributed testing.
> I suggest that we change it to fail on failed tests instead of
> changing the behavior of the current test mechanism.
I'm not sure I understand what you mean. My patch does not change any
behavior of the current test mechanism. I will not introduce any such
changes, not even if you wanted me to. :-) So that sounds like a false
I can change the new run-test system to fail on failed test. To keep it
as readable as it is now, I would need to make a hack with the top-level
FailureHandler. This sure can be done, but increases the risk of
unintended consequences. But since you feel so strongly about this, I
assume that's the way to go.
> There will likely be other changes needed before automated systems can
> use this, but this is a very fundamental part of the API. If make
> doesn't fail, any wrapping tool/script/system is unable to know if the
> run was successful. All other build systems I know of that run tests
> do this (gradle, maven, ant etc).
More information about the build-dev