[rfc] [icedtea-web] blacklist for reproducers
jvanek at redhat.com
Tue Apr 10 04:42:48 PDT 2012
On 04/09/2012 12:56 AM, Omair Majid wrote:
> On 04/06/2012 08:56 AM, Jiri Vanek wrote:
>> This is second from total of 5 patches which are introducing Tets For
>> Thus particular one is introducing blacklist. The test-engine is
>> delivered to class-path together with all reproducers' testcases. If the
>> file form engine is interface, have no @Test methods or does not meet
>> constructor requirements, then the file fails, and this is making our
>> statistics misleading.
>> Also I'm expecting growing of engine (now I'm planing @bug and @browser
>> annotations and some more logic around. See content of ignore-list after
>> all this changes. Also I would like to refactor ServerAccess class
>> little bit in long term (as it have grown littlebit and have to much
>> inner classes right now) future...
>> Probably cleaner solution will be to separate testengine from testaces
>> more then it is now, but I have tried this refactoring and I was getting
>> deeper and deeper and it was bringing more and more issues. From this
>> I'm really for blacklist rather then for refactoring.
>> Also one more benefit is here - you can easili blacklist all reproducers
>> except one or two fro make ;)
I was afraid you will say something like this...
> I feel rather uncomfortable with this. I really think the underlying
> problem should be fixed instead. Using band-aids like this works in the
> short-term but makes icedtea-web worse in the long term.
> If there is some effort required to fix the test-engine, lets do that.
> If the tests are wrong, lets remove them. If it makes sense to run only
> a few tests, adding that would be pretty nice (like how jtreg does it by
> allowing a user to specify a directory to find tests in) but a blacklist
> looks really fishy.
So lets dig what is necessary to do.
netx directory with sources of nex
│ └── jnlp
│ ├── tools
│ └── util
and we have
tests directory with
├── jnlp_tests - source files and testcases for reproducers
│ ├── signed
│ └── simple
├── junit-runner - launcher for both testsuites
├── netx - directory containg unittests, pac tests, *infrastructure for reproducers* and *tests of
infrastructure* (virtual server...)
│ ├── jnlp_testsengine - *both* infrastructure for reproducers *and* tests of infrastructure
│ │ └── net
│ │ └── sourceforge
│ │ └── jnlp
│ ├── pac - sources of pac tests
│ └── unit - sources of unittests
│ └── net
│ └── sourceforge
│ └── jnlp
│ ├── application
│ ├── runtime
│ ├── templates
│ └── util
│ └── replacements
The we have compile-time and run-time dependencies in makefile (I'm usinf sources locations instead
of tests.build in this text wilfully)
They are currently :
compile-time of unittests:
netx and tests/netx/unit
run-time of unittests:
junit-runner netx and tests/netx/unit
compile-time of reproducer's sources:
netx and tests/jnlp_tests/signed|simple/reproducer/src/
compile-time of reproducer's testcases:
netx tests/netx/jnlp_testsengine and tests/jnlp_tests/signed|simple/reproducer/testcase/ (all
runtime-time of reproducer's sources:
junit-runner netx and from its jnlp/html (mostly in jnlp_tests_server with jars created from
run-time of reproducer's testcases:
junit-runner netx tests/netx/jnlp_testsengine and
tests/jnlp_tests/signed|simple/reproducer/testcase/(all testcases together)
And second thing is where the classes of tests are taken:
for unittests all files inisde tests/unit
for reproducers all files tests/netx/jnlp_testsengine and
The error is obvious. tests/netx/jnlp_testsengine contains both tests and logic
My opinion here will be to unify above as much as possible.
When you will check  for browserTests-BugAnnotation  you will see how few other issues "solved
1) move all tests from tests/netx/jnlp_testsengine to tests/netx/unit where they belongs IMHO
1.5) With this refactoring I would like to extract all inner classes from ServerAccess
2) fix compile and runtime class-path of unittests for tests/netx/jnlp_testsengine
3) during reproducers run launch *just* compiled testcases
4) no more tests in tests/netx/jnlp_testsengine, framework will be tested in unittests (same as (1)
but just for ensuring..:) )
4.5) unittets and reprodyucers runs should have same classapth in runtime/compiletime. => unify them
in some variable(s)(?)
5) I'm also for adding tests/netx/jnlp_testsengine to compile class-path of junit-runner (as it
looks useful from [1,2] point of view.
6) Maybe after all of above done the jnlp_testsengine can be renamed and moved somewhere else. (and
also jnlp_tests probably should be moved to tests/netx/) But This should be done as separate changeset.
1-6 - ServerAccess api should not be changed, although maybe some methods private/public/protected
statuses will be transformed due to moving of tests)
What do you think?
For launching just some tests I would like then introduce some whitelist. The logic will be same as
was introduced in this discarded patch, but default value will be ".*" Before wanted run it will
be possible to enter regex matching the set of reproducers determined for run.
What do you think?
More information about the distro-pkg-dev