Questions about pre-submit testing using GitHub Actions
robin.westberg at oracle.com
Mon Oct 12 07:16:55 UTC 2020
> On 9 Oct 2020, at 17:47, Volker Simonis <volker.simonis at gmail.com> wrote:
> first of all thanks a lot for adding support for pre-submit testing
> using GitHub Actions. I think this is a really useful feature!
> Unfortunately, I couldn't find much documentation about it except some
> emails on this list and the initial PR , so I still have a few
> questions :)
Thanks! We’re working on documenting it a bit better on the Skara OpenJDK wiki page which should be ready for publication soon. :)
> 1. All the pre-submit test that have been run for me until now (e.g.
> ) have failed for "Windows x64 (hs/tier1 gc)" presumably because of
> a generic "out of disc space" error:
> Error: There is not enough space on the disk. :
> This starts at "TEST: gc/shenandoah/compiler/TestClone.java#id5" after
> which all subsequent tests fail because of insufficient disc space:
> error: error while writing TestClone: There is not enough space on the
> 2020-10-09T12:03:09.6475592Z TEST RESULT: Error. Write to temp. JTR
> file failed (old JTR intact):
> Is this a known problem? Is somebody working on fixing it?
Yep, this was fixed by https://bugs.openjdk.java.net/browse/JDK-8253778. If you are still seeing these errors in a PR, you will most likely want to merge the latest upstream “master” into your branch so that the fixes gets applied.
> 2. When looking at "Linux x64 (hs/tier1 gc)" Ican see that there are
> 44 test failures:
> 2020-10-09T11:40:16.8365547Z Test results: passed: 255; failed: 44
> 2020-10-09T11:40:22.0971166Z ==============================
> 2020-10-09T11:40:22.0973305Z Test summary
> 2020-10-09T11:40:22.0973969Z ==============================
> 2020-10-09T11:40:22.0974393Z TEST
> TOTAL PASS FAIL ERROR
> 2020-10-09T11:40:22.0974844Z >> jtreg:test/hotspot/jtreg:tier1_gc
> 299 255 44 0 <<
> 2020-10-09T11:40:22.0975267Z ==============================
> But the test group has a green check mark and is flagged as passed in
> the report: "Linux x64 (hs/tier1 gc) succeeded (3 annotations)". How
> can this happen? Is this a known problem?
> By the way, the 44 failing test all fail on Windows and MacOS as well
> because of a crash in:
> 2020-10-09T12:03:06.0162304Z # Internal Error
> pid=6344, tid=3700
> 2020-10-09T12:03:06.0163057Z #
> assert(Thread::current()->is_VM_thread()) failed: Must be
> It looks like the framework only recognizes jtReg "Errors" but not
> "Failures" as errors.
Yep, this was a problem with the GitHub Actions definitions themselves, and was fixed by https://bugs.openjdk.java.net/browse/JDK-8253865. But again, as the jobs run with the version of the definitions that are present in your branch, you’ll need to refresh it for the fix to be applied.
> 3. Would it be possible to save the hs_err files along with the .jtr
> files in the case of crashes? I think this would be extremely useful
> for quickly diagnosing problems.
Sure, that makes sense! Filed https://bugs.openjdk.java.net/browse/JDK-8254561.
> 4. I don't exactly understand the "free tier" you mention in your PR
> . From the GitHub actions side as described in their "Usage limits"
> documentation  it seems we're on the safe side. But that
> information somehow contradicts with the information about "GitHub
> Free for user accounts"  which mentions that free user accounts
> only have 2000 free GitHub Actions minutes per month. In addition
> "About billing for GitHub Actions"  mentions that these free
> minutes are subject to a multiplier which is one for Linux, two for
> Windows and ten for macOS which effectively means that after running
> 200 minutes on macOS our personal free minutes for a month will
> already be consumed. So to me it seems that GitHub Actions "Usage
> limits" documentation  only sets an upper bound for job and
> workflow execution time of Actions but this has to be interpreted in
> the context of the totally available, free minutes for Actions. But
> maybe I'm wrong?
> If you know better :) could you please explain which of these limits
> apply and how many automatic tests jobs they are approximately
The pricing and usage documentation is indeed a bit confusing as it is split in several parts. The part that is relevant for us is slightly higher up in the link you referenced as  - https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#about-billing-for-github-actions - which says:
"GitHub Actions usage is free for public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account.”
So the limits and billing you see in  and  refers to what you can use with private repositories if you have a free account, which I guess most of us have. But public repositories like your fork of the JDK are subject to the limits in  which basically means that you can run up to 20 jobs in parallel all day long.
> Thank you and best regards,
>  https://github.com/openjdk/jdk/pull/284
>  https://github.com/simonis/jdk/runs/1230911240
>  https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#usage-limits
>  https://docs.github.com/en/free-pro-team@latest/github/getting-started-with-github/githubs-products#github-free-for-user-accounts
>  https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions#about-billing-for-github-actions
More information about the skara-dev