RFR: 8041151: More concurrent hgforest.
mike.duigou at oracle.com
Thu Apr 24 23:57:27 UTC 2014
On Apr 22 2014, at 01:27 , Staffan Larsen <staffan.larsen at oracle.com> wrote:
> I ran the patch on OS X and it worked there too. Thanks!
> You mention “status code from subprocesses” which got me thinking of a problem I frequently run into when I have some uncommitted changes in one of the repos:
> .: searching for changes
> .: adding changesets
> .: adding manifests
> .: adding file changes
> .: added 1 changesets with 2 changes to 2 files (+1 heads)
> .: not updating: not a linear update
> .: (merge or update --check to force update)
> This output is frequently lost to me in all the other output, would it be possible to catch these errors and somehow highlight the failure at the end of the run?
Unfortunately we don't buffer the output so there's no way at exit time to go back and gather the output.
The good news is that the planned changes should be directly helpful for the case you are encountering. We/I hope to add rebase support to get_source.sh so that local changes whether unpushed changesets, mq patches or uncommitted changes will be rebased to the new tip whenever you pull.
> I guess I could go spelunking in the code to find out, but I’m too lazy and prefer to bother someone else. :)
Not a problem.
> On 19 apr 2014, at 01:21, Mike Duigou <mike.duigou at oracle.com> wrote:
>> Hello all;
>> This is an improvement to hgforest to increase it's concurrency behaviour. Currently hgforest.sh limits the rate at which it starts new sub-processes because it wants to limit the number of concurrent tasks. The naive approach it takes can cause unnecessary delays. For sequences of operations that are entirely local the overhead of waiting is significant. The revised implementation uses fifos for completion notification on capable platforms or compares started task count to completed task count in a shorter sleep loop.
>> The intention is to use the enhanced concurrency to allow for a fancier get_source.sh that can handle rebasing for mq patches. This involves running a half dozen commands through hgforest. With my current in-development get_source.sh script changes these hgforest changes provide a 10X speedup. (4s vs 40s)
>> The changeset also incorporates a build-dev suggested improvement to extra base repo url handling and other minor fixes (status code from subprocesses).
>> JBSBUG: https://bugs.openjdk.java.net/browse/JDK-8041151
>> WEBREV: http://cr.openjdk.java.net/~mduigou/JDK-8041151/0/webrev/
>> I've so far tested it on successfully on Linux (Ubuntu 13.10) Solaris (10u9) and Cygwin x64
More information about the build-dev