Reviving JEP 230: Microbenchmark Suite
claes.redestad at oracle.com
Thu Sep 6 15:16:08 UTC 2018
JEP 230: Microbenchmark Suite was proposed during JDK 9 development,
but was put on hold for various mundane reasons. I think time has come
to revive this effort, and I've volunteered to take over ownership of
Some time after JEP 230 was temporarily abandoned, the
jmh-jdk-microbenchmarks project was conceived to ensure some of the
work done to prepare didn't go to waste. A set of previously closed
source microbenchmarks was open sourced and contributed there, and
artifacts from this project are actively being used to track
performance. This situation is not ideal, however.
A side-effect of the more rapid release cadence is that the bulk of new
feature development is being done in project repositories. This creates
some very specific challenges when developing benchmarks out-of-tree,
especially those that that aim to track new and emerging APIs.
For starters we're pushed toward setting up branches that mirror the
project layout of the various OpenJDK projects (valhalla, amber, ...),
then we need to set up automated builds from each such branch, then have
some automated means to match these artifacts with appropriate builds of
the project-under-test etc. And how do we even deal with the case when
the changes we really want to test are in javac? Nothing is impossible,
of course, but the workarounds and added work is turning out to be costly.
By co-locating microbenchmarks, matching the JDK-under-test with an
appropriate microbenchmark bundle would be trivial and automatic in most
cases, and while one always need to be wary of subtle changes creeping
into benchmark bundles and the JDK between builds, this is something we
already test for automatically as regressions are detected.
Also, we wouldn't really lose the ability to pick up an existing
microbenchmark artifact as appropriate (a "golden" bundle; appropriate
when comparing library and VM changes over a longer time). A standalone
project can be considered a good enough fit for that case, so one
alternative to moving all of jmh-jdk-microbenchmarks into the JDK would
be keep maintaining the standalone project for benchmarks that are
considered mature and stable. My preference is to make
jmh-jdk-microbenchmarks redundant and then shut it down, though. I think
most would typically build and keep a "golden" bundle of a specific
version around for longer-term regression tests, so a separate
standalone project of JDK micros doesn't make much sense.
Also of note is that one of the more controversial aspects of JEP 230
was the question of where to put them. We've since consolidated the
OpenJDK source repositories into a single one, so much of what was
discussed back then no longer applies. Mainly this simplifies the steps
needed to integrate a microbenchmark suite into the JDK, and I've
updated the JEP text accordingly.
More information about the jdk-dev