jtreg testing integrated
Jonathan.Gibbons at Sun.COM
Tue May 20 09:56:51 PDT 2008
On May 20, 2008, at 2:32 AM, Mark Wielaard wrote:
> Hi Martin,
> On Mon, 2008-05-19 at 08:30 -0700, Martin Buchholz wrote:
>> On Mon, May 19, 2008 at 7:56 AM, Andrew John Hughes
>> <gnu_andrew at member.fsf.org> wrote:
>>> 2008/5/19 Mark Wielaard <mark at klomp.org>:
>>>> make jtregcheck -k runs the testsuites of hotspot (4 tests, all
>>>> langtools (1,342 PASS, 1 FAIL - the version check) and jdk (2,875
>>>> of which about 130 fail - rerunning tests now). corba, jaxp and
>>>> don't come with any tests. This takes about 3 hours on my machine.
>> Once upon a time, I wrote a test that made sure the hotspot
>> and jdk library's idea of the current version and supported targets
>> were in sync. Unfortunately, it is not a requirement on hotspot
>> integrations that they pass this test, so the test starts failing
>> hotspot starts supporting the class file version number for the next
>> major release. At least this is a strong hint to the javac team to
>> catch up soon by incrementing their supported targets, etc...
> In this case it is a more mundane version check failure:
> javac thinks its version isn't 1.6.0, but javac 1.6.0-internal
>> I like a policy of "Read my lips; no new test failures" but OpenJDK
>> is not quite there; we get test failure creep when changes in
>> one component break another component's tests.
> Yes, that would be idea. At least for openjdk6/icedtea we seem to be
> pretty close actually. It will be more challenging for openjdk7. I
> haven't quite figured out all the dynamics around "workspace
> integration". But I assume we can get the master tree to zero fail and
> then demand that any integration cycle doesn't introduce regressions.
>> The jtreg tests were originally designed to be run only by Sun JDK
>> development and test engineers. If someone can come up with a
>> portable way of testing network services (like ftp clients) without
>> setting up a dedicated machine with a well-known name, that
>> would be good. Alternatively, making the name of this
>> machine configurable when jtreg is run would also
>> be an improvement, and a much simpler one. But the obvious
>> idea of using environment variables doesn't work. Most environment
>> variables are not passed to the running java test program.
> Making it configurable, or even ignorable with keywords would be
> for distribution testing. Most distributions don't allow their build
> daemons to access the network. But for quality control it is essential
> that they do run the full test suite.
jtreg allows tests to be tagged with keyords, that can be used on the
line to filter the tests to be executed.
> I haven't made an inventory of what services would be needed on a
> machine to be replace javaweb.sfbay.sun.com with a public machine, but
> we can certainly run some services on icedtea.classpath.org or maybe
> of the openjdk.java.net machines.
>> If it's considered acceptable for IcedTea hackers to get their
>> hands dirty with not-100%-free technology, y'all could try
>> running the jtreg tests against IcedTea, vanilla OpenJDK7,
>> OpenJDK6, and JDK 6u6, and comparing the test failures.
> :) Well, the whole idea behind IcedTea is to provide an OpenJDK
> derivative that doesn't depend on any non-free build or runtime
> But I am certainly interested in comparing results. I do think
> and IcedTea are now so close that we shouldn't be seeing any test
> differences between the two.
> That brings up the question how to export a JTreport/JTwork
> I only posted the text results http://icedtea.classpath.org/~mjw/
> since the JTreport and JTwork files have all kinds of hard coded
> absolute path references. It would be nice to be able to export it all
> so I can upload it to some public site for others to look at and
At a minimum, you'd want to publish the summary.txt files from the
Note also that JT Harness comes with a couple of servlets you can
install for pretty
viewing .jtr and .jtx files.
More information about the compiler-dev