RFR: JDK-8146403: Windows build can be faster
markus.gronlund at oracle.com
Tue Jan 5 11:09:45 UTC 2016
(not a review)
Thanks a bunch Erik for taking this on, +1.
The Cygwin abstraction layer is just painfully slow - as you point out, it has to do with spawning new process all the time (where the Windows architecture expects fat processes using many threads instead of short lived Posix processes/threads).
Just the amounts of spawned /usr/bin/find.exe in directories containing a lot of files is proportionally slow to the many files contained therein (could be an optimization hint to split files into several directories perhaps).
Any work here to help speed this up is very much appreciated.
From: Erik Joelsson
Sent: den 5 januari 2016 11:58
Subject: RFR: JDK-8146403: Windows build can be faster
During the hotspot makefile conversion, we have been reminded of inefficiencies when running make in Cygwin. We still have a pretty severe performance regression in the new hotspot build compared to the old on Windows in certain situations, my laptop being one such situation. A recent comparison there between just the old and new hotspot build:
release new 00:04:30
release old 00:03:10
fastdebug new 00:04:59
fastdebug old 00:03:37
Much of the extra time is spent spawning various shell processes for book keeping, like saving failure logs, creating header dependency files etc.
It also takes a very long time to do a "do nothing" rebuild when nothing has changed. On my laptop, repeating "make jimages" often takes as long as 40 seconds to figure out that nothing needs to be rebuilt. In this case there are several culprits.
I have been working on improvements in these areas to reduce the overhead. The "do nothing" rebuild of jimages is down to around 25 seconds. A full images build is around 1-2 minutes faster from 24 to 22 minutes, but fluctuates quite a bit. The new hotspot build is also
Here is a list of the kinds of changes I've made:
* Rewrote logger.sh to use a different construct. It drastically reduces the number of bash processes being spawned. Basically you can pipe like
this: "> >(tee logfile) 2> >(tee logfile >&2)". I also managed to reduce the extra sed/grep commands needed for the header dependencies even more. A side effect of this is that the log files for each native compilation unit are always saved instead of being deleted on successful compilation. I see no issue with this.
* Changed all recipes that contain echo for logging to instead use new LogInfo macro, which in turn calls $(info ) if appropriate.
* Changed all recipes that contain mkdir to instead use MakeDir macro, which only executes mkdir if the directory doesn't exist.
* In HotspotWrapper.gmk there is an optimization to avoid calling the hotspot makefiles at all if nothing has changed. This runs a find over the whole hotspot repository. I reduced this to only run over src and make sub dirs to avoid the pretty large test dir (since test/closed has grown quite a bit).
* Split Tools.gmk into CompileTools.gmk and Tools.gmk, to avoid having the buildtools compilation being reevaluated by all makefiles needing the tools definitions.
* Split Modules.gmk to avoid having the module deps generation being reevaluated multiple times. Made the new GenerateModuleDeps.gmk an explicit call from Init.gmk.
Since these improvements affect much more than just the new hotspot build, I intend to push this to JDK 9 directly.
More information about the build-dev