Proposal to revise forest graph and integration practices for JDK 9
joe.darcy at oracle.com
Sat Nov 23 08:34:26 PST 2013
The current arrangements of sets of integration forests for a JDK
platform release, like JDK 8, impose high overheads on development. I'm
proposing we use an alternate forest arrangement for JDK 9 that will
dramatically reduce the propagation time of fixes across the set of
forests. More details below.
JDK release projects for new Java SE platforms, like JDK 8 for Java SE
8, have long used a graph of forests structured roughly as follow:
* A master forest for the release
* A thicket of integration forests for particular teams or technology
areas. Today in JDK 8, the "TL" forest hosts changes in tools related to
javac as well as core libraries. Forests for various client libraries
(2D, awt, swing) host changes in those areas. A HotSpot forest, fed in
turn by several HotSpot team forests, hosts VM-related changes.
Generally, each integration forest only accepts changes to a subset of
repositories. For example, the HotSpot-related forests typically only
accept changes to the hotspot repository. The TL forest accepts changes
to library-related repos (jdk, jaxp, jaxws, etc.) and langtools, but not
After some amount of testing and other verification steps, changes in
one integration forest are integrated into master, typically on a weekly
or bi-weekly basis. From master, the fix then propagates down to
integration forests according to the policies of that forest. A fix
could be in master for several days or more before being propagated to a
particular integration forest.
While this structure has provided a great deal of cross-team isolation,
it has come at the cost of high propagation delays of fixes to all
forests. This propagation delay combined with only pushing fixes to a
subset of repos also severely complicates making coordinated changes
which span across repositories, as often occurred in Project Lambda and
which chronically occurs for technologies like servicability. To give a
representative example, consider a hypothetical change which requires
updates to both the HotSpot runtime area as well as core libraries. If
the change is first pushed to HotSpot, the propagation proceeds like:
fix pushed to HotSpot runtime forest -> HotSpot main -> JDK 8
master -> TL -> core libs engineer's forest
At this point, the libraries half of the change can be pushed:
core libs fix pushed to TL -> JDK 8 master -> HotSpot main ->
HotSpot runtime -> HotSpot runtime engineer's forest
This cycle to separately push both halves of what is conceptually a
single fix and wait for them to propagate can take about four weeks.
(Worse, often there needs to be a third push to complete the fix since
the first push is often to "accept old way or new way" and the third
push updates this to "accept new way only." When such a clean-up push is
needed, it takes another two weeks to fully propagate.)
Since more projects requiring cross-repository coordination are expected
in the future, I'm proposing a number of changes to the forest structure
and management policies in JDK 9 to reduce the propagation delays.
* A master forest that is a time-delayed version of dev; dev described
* The dev forest conceptually replaces TL and hosts all
libraries-related changes. When the sources in dev are in a known-good
state, that state can be integrated to master. This integration cycle
would happen at least weekly. In a change from current practice, HotSpot
changes would be integrated into dev and *not* into master. All other
team forests would also integrate into dev rather than master.
* Coordinated HotSpot + other component fixes would *both* get first
pushed through the HotSpot forest. From hotspot the full fix would be
integrated into dev. If additional testing was appropriate for the
non-HotSpot fix, that testing should occur before integration into dev.
* Regular promoted builds based on master would continue.
By having team forests integrate directly into dev as well as having
many libraries developers pushing directly to dev, the dev forest serves
as an active collaboration area with greatly reduced propagation times
across the whole system. With this model there is less cross-team
isolation; teams and individuals are responsible for promptly fixing any
breakage which is introduced. If problems are not quickly addressed, a
problematic changeset may be anti-delta'ed.
(Conceptually, in this model a separate master forest is not strictly
needed since the known-good states could be indicated using a mechanism
like Hg tags. However, while adjusting to the new model and to allow for
fixes directly to master in exceptional circumstances, I'm proposing a
physically separate master forest be retained. The distinct URL of
master will also clearly indicate known-good states of the source code.)
Please send comments on the above by November 29.
More information about the jdk9-dev