Adding Microbenchmarks to the JDK forest/trees (JEP-230)
staffan.friberg at oracle.com
Fri Dec 5 19:46:38 UTC 2014
On 12/04/2014 10:23 PM, joe darcy wrote:
> On 12/4/2014 4:34 PM, David Holmes wrote:
>> Hi Staffan,
>> On 2/12/2014 10:08 AM, Staffan Friberg wrote:
>>> Hopefully this is the right list for this discussion.
>>> As part of adding Microbenchmarks to the OpenJDK source tree, I'm
>>> to understand how we best would add the benchmark sources to the
>>> existing OpenJDK tree structure.
>> Is there a reason this needs to be inside the OpenJDK instead of a
>> stand-alone project? If it ends up in its own repo and the repo is
>> not needed by anything else, then it is already like a stand-alone
> I think David is raising a good question here. A related question is
> if we want to update/fix the microbenchmarks, in how many release
> trains will we want to make that fix? If the expected answer is much
> greater than one, to me that would seem to me to argue for a separate
> forest in the overall OpenJDK effort, but not bundled into a
> particular release.
> For example, in the past, the webrev.ksh script was included in the
> JDK forest. That was an improvement over every engineer having his or
> her personal fork, but still made keeping webrev updated unnecessarily
> difficult since any changes would need to be done multiple time and
> there is nothing fundamentally binding a version of webrev to a
> particular JDK release. Likewise, even though (to my knowledge) jtreg
> is only used for testing the JDK, jtreg has its own repository and
> release schedule separate from any JDK release.
> So if the microbenchmarks are to a first-approximation version
> agnostic, then they should probably have a forest not associated with
> a JDK release. If they are tightly bound to a release, that argues for
> putting them into the JDK forest itself.
The reasons are similar to co-locating functional tests, stable test
base that is not moving after FC, tests support new features and all
work with the JDK you are developing.
Looking at some of the goals in the JEP and why I think co-locating helps.
* Stable and tuned benchmarks, targeted for continuous performance testing
o A stable and non-moving suite after the Feature Complete
milestone of a feature release, and for non-feature releases
To fulfill this having a separate repository would force us to branch
and create builds in sync with the JDK we want to test. As the number of
JDK version increases so will the microbenchmark suite. Making it
complex for anyone adding or running tests to keep track of which
version one should use for a specific JDK branch.
As an example, for a stable branch we probably do not want to update the
JMH version, unless there is a critical bug, as doing so can invalidate
any earlier results causing us to lose all history that is critical when
tracking a release. However during development of a new release we
probably want to do that to take advantage of new features in JMH. And
we might require new features in JMH to support changes in the JDK.
Fixing bugs in microbenchmarks will in a similar fashion to any bugs
sometimes need to be backported, but we would have similar struggles in
a separate repository if we want to keep the suite stable for multiple
releases and have multiple branches to support that.
JMH similar to jtreg is a separate repository already, and will stay
that way. The co-located part would be the benchmarks, which similar to
the tests written using jtreg benefit from being co-located.
o Easy to add new benchmarks
o Easy to update tests as APIs and options change, are deprecated,
or are removed during development
o Easy to find and run a benchmark
Having it co-located reduces the steps for anyone developing a new
feature to push benchmarks at the same time as the feature in the same
way as functional tests can be pushed.
Any benchmark relying on a feature being removed can easily be deleted
without concern about reducing the test coverage of older releases still
Having it in a completely separate repository I fear will cause
microbenchmark to be out of sight and out of mind, which is the opposite
of the intention of creating the suite. We want more people to write and
run benchmarks in a simple way. Co-location will allow anyone to simply
just add and do make to build for their current branch and test the
feature they are working on, and those benchmarks will directly be
picked up for regression testing once pushed.
Hope this helps to further clarify the reason for having the suite as
part of the OpenJDK source tree.
More information about the build-dev