[External] : Re: Clarification regarding the GitHub Actions pre-submit testing
stuart.marks at oracle.com
Tue Mar 23 06:50:44 UTC 2021
On 3/19/21 2:02 AM, Aleksey Shipilev wrote:
> And here I thought I successfully avoided the word "must" to avoid this impression.
> To me more precise, I would say "GHA should be green before integration", where
> "should" is as per ISO definition: "should" indicates a recommendation, where "a
> recommendation is defined as an "expression [...] conveying a suggested possible
> choice or course of action deemed to be particularly suitable without necessarily
> mentioning or excluding others."
> After yesterday's discussion, that can be amended to "You can reasonably ignore GHA
> failures by securing the reviewers agreement that: a) failures are not related to
> the current PR; or b) failures, while related, are beyond the scope of this PR
> and/or contributor capabilities, and the breakage is, while unfortunate,
> intentional, known and recorded in these follow-up bugs".
> I think that is a reasonable thing to do, process-wise: failures do not necessarily
> block integration, simple failures are detected and hopefully fixed before
> integration, complex failures that require followups become known right away.
OK, I'm glad to hear there's some pragmatism here. What you're saying is quite
Your nuanced statement, while generally agreeable (at least to me) still allows a
situation where disagreements over expectations and responsibilities can occur. I
think it's fairly important to be explicit about these. The reason is that while I
think we can generally assume that most people on here are reasonable, it's not
valid to assume that reasonable people will all come to the same conclusion on
What makes this complicated is that the GHA submit actions aren't just one thing.
There's a whole bunch of stuff, with lots of different configuration options about
what gets run on which platform. It's important to make sure that decisions about
these configurations, and changes to them, are discussed and decided openly, and
that adjustments to expectations and responsibilities are also communicated accordingly.
Let me take a hypothetical example of enabling tier2 tests in GHA. (Just to be
clear, nobody has proposed this. This is something I just made up.) We might want to
do this in order to improve test coverage and to provide earlier warning of actual
failures, especially to people outside of Oracle. However, the tier2 tests include
networking tests, which are notoriously dependent on machine configurations,
timeouts, etc., and thus are susceptible to intermittent failures. A consequence of
this change would then be that the GHA board would have red spots on it more of the
If there's open discussion of this, and everyone is clear that getting better test
coverage is worth tolerating more red on the GHA board, then that's ok. If the
decision is NOT to enable tier2, because keeping the GHA board green is more
important than improving test coverage, that's ok too.
What I do NOT want to see is a situation where a few people get together and decide
to make some change, and this is a surprise to the rest of the project. This leads
to mismatches in expectations and responsibilities. I could see the following
- Hey, why am I suddenly getting red spots on the GHA board?
- Those are intermittent failures from tier2.
- When did we start running those?
- The other day; I think there was a message on build-dev about it.
- Well I can't integrate since the board isn't green; what am I supposed to do now?
- Oh, just ignore the intermittent failures, they're ok.
- When did that rule change?
- There was no rule change, it's accommodated by the policy expressed in the
middle of some email from Shipilev buried in the middle of a thread on jdk-dev
The way to avoid such problems is to have clear communication about changes to the
GHA submit actions, especially those that change responsibilities of all JDK
developers. (This would be a good topic for a Committers' Workshop, but as we know,
circumstances have prevented these from occurring.) Meanwhile, email to jdk-dev will
have to suffice.
More information about the jdk-dev