[icedtea-web] RFC: disable tests that are known to fail
jvanek at redhat.com
Thu Mar 22 09:28:21 PDT 2012
On 03/22/2012 04:14 PM, Omair Majid wrote:
> On 03/22/2012 04:13 AM, Jiri Vanek wrote:
>> To be clear - I'm for some kind of skipping, but do not like the idea of
>> used @ignored annotation.
> Assume that most developers are not running the junit tests through the
> command line but using their ide's junit integration. Any ideas on how
> skipping can be accomplished? (Without configuring their ide). We want a
> default 'make check' (and whatever method an ide might use to run the
> junit tests) to show that all tests have passed.
Eg I'm not doing so. I'm running in ide just suite or two which I expect to be affected by my work, and then run make all-possible-test and check reports/coverage.
But yes, from this point of view own annotation is not an solution.
> To look at it from the other point of view, why dont we have a flag/env
> var that says ignore the @Ignore annotation during make check (or
> something like that)?
ha! Nice reversed idea :) In this case My opinion is to have configure-able variable to ignore @ignore ( :D ) during make.
(As I'm thinking about implementation of ignore @Ignore ... it can be tricky!)
But when this reversed idea is connected with the "preferring ide" mentioned above, hmhm my-mind-is-deadlocked.
I'm still more preferring own annotation:
- in ide it will fail, developer will check, why the hell, will see all the reasons (eg that it is representing an issue) and ignoring through rest of his patching.
- After his patch is done, he will return to the discovered known-to-be-failing test and will make new patch :)
- will run make check/distcheck both with enabled/disabled "ignore KnownToBeFailing" and so will know all the impacts
- in ide will run testsuite, will see that nothing failed, something was ignored, will never check what and will not care why...
- will see both in ide and make all-tests that everything is passing, something is ignored (but who cares!)
- will never be "forced" to discover amazing --ignooreIgnore configure option :)
I'm definietly for marking known-too-be-failing tests, but whether own annotation or Ignore (with ignore @ignore) I'm 50/50 now .
More information about the distro-pkg-dev