Refuse a required context and a check name that disagree (#71) - #131
Merged
Conversation
The required set holds strings and the workflows declare strings, and nothing on the platform ties the two together. A job renamed in passing removes itself from the required set while the tab still looks green, and a context required under a name nothing reports blocks every merge without saying why on the pull request. Five issues in the plan carry that warning in different words and nothing acted on it. internal/contexts is the comparison. It is a function over two lists and reads nothing, so every rule is proved against a fixture written out in full. cmd/contexts is the entry point, and .github/workflows/contexts.yml is where the ruleset is fetched, because half of this is a live setting rather than a file and lab opens no network connection. The deliberate absences are a list beside the comparison rather than a paragraph in a document. Most of it is one fact repeated: the ruleset requires no status check at all today, so every name this tree declares is outside the set, and each is written down rather than the comparison switching itself off while the set is empty. That is what catches the rename now: an absence names a literal string, so a job renamed while the set is still empty leaves its entry pointing at nothing and the check goes red. Two of the refusals also run without a network, in the ordinary suite, so a rename reddens the machine of whoever made it rather than waiting for the pull request. What a green result here does not say is written where the comparison is defined: it cannot judge behaviour, and it reads the ruleset as the API answered on the day it ran. The means is Go, which is what every other check in this tree is written in and what record 0001 chose. The fetch is the one thing that is not Go, and it is held to a single gh call in the workflow because a ruleset is not in the checkout. Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
The first commit read check names out of the workflow files and nothing else,
which is short by two. A code-scanning upload creates a check run of its own,
named after the analysis rather than after the job that uploaded it, and it is a
name a ruleset can require exactly like any other. Measured on the head commit
of this branch rather than supposed:
gh api "repos/Flowfin/lab/commits/$(git rev-parse HEAD)/check-runs" --paginate \
--jq '.check_runs[] | "\(.name)\t\(.app.slug)"' | sort -u
Twenty-four names, of which twenty-two come from github-actions and two, CodeQL
and zizmor, come from github-advanced-security.
Without this the comparison would refuse either of those as a required context
nothing reports, which is a false red on the change that assembles the required
set, and a false red is the failure this check exists to avoid rather than to
cause. The two names are written down beside the absences with the command that
found them, and what that costs is stated there: the string they report under is
decided by the analysis rather than by a line in this tree, so a change to it is
not caught here. The report says which of the two claims a reader is holding.
Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #71
What this changes
Adds the comparison between the two lists that have to agree for the merge gate
to mean what a reader thinks it means: the contexts the ruleset requires on the
default branch, and the check names the workflows in this tree declare.
internal/contextsis the judgement. It is a function over two lists and readsnothing, so every rule is proved against a fixture written out in full rather
than against whatever the machine running the suite happened to contain.
cmd/contextsis the entry point and takes the required set on standard input..github/workflows/contexts.ymlis where the ruleset is fetched, with aread-only token granted on that one job, because half of this is a live setting
rather than a file in the checkout and
labopens no network connection.Five refusals, each with a fixture that trips exactly it and no other:
deliberate absence
resolve, so a file in a shape this reader was not built for reddens the check
instead of quietly declaring the wrong string
The deliberate absences are a list beside the comparison in
internal/contexts/contexts.gorather than a paragraph somewhere else, with areason on every entry and the issue that retires it where the absence is not
permanent. One entry is permanent: the supply-chain self-audit publishes from
the default branch and has no pull-request trigger, so requiring it would
require a context that never arrives on the thing being gated.
The rest of the list is one fact repeated. The ruleset requires no status check
at all today, so every name this tree declares is outside the set, and each one
is written down rather than the comparison switching itself off while the set is
empty. That is what catches a rename now: an absence names a literal string, so
a job renamed while the set is still empty leaves its entry pointing at nothing.
Issue #26 retires those entries by moving the names into the ruleset.
Two of the refusals also run in the ordinary suite, with no network, because the
answer is knowable without asking while the required set is empty.
Not every check run comes from a job, and the first commit here missed that. A
code-scanning upload creates a check run named after the analysis rather than
after the job that uploaded it, so it is a name a ruleset can require and no
reader of the workflow files can ever find. Measured rather than supposed:
On this branch's head that printed twenty-four names, of which twenty-two come
from
github-actionsand two,CodeQLandzizmor, come fromgithub-advanced-security. Those two are written down beside the absences withthe command that found them. Without that the comparison would refuse either of
them as a required context nothing reports, which is a false red on the change
that assembles the required set.
What failure it prevents
A check name is what the required set refers to and nothing on the platform ties
the two together. A job renamed in passing removes itself from the required set
while the tab still looks green, which is the quieter of the two failures and
survives indefinitely. A context required under a name nothing reports blocks
every merge and gives no reason on the pull request, so the first response is to
wait and the second is to drop the context, which repairs a blocked board by
making the gate smaller.
What was run
At the commit being pushed, on
windows/amd64.Each of the five refusals was deleted in turn and the suite re-run. All five
leave it red:
The rename, against this tree rather than a fixture. Renaming the
vetjob inbuild.ymltogo vetand leaving the required set alone:The other direction, through the command, with three contexts fed in and the
last of them naming nothing:
The whole thing against the live ruleset, which is what the workflow does:
The two refusal messages above are elided at the point where they repeat the
reason the absence list carries, which the run prints in full.
The means is Go, which is what every other check here is written in and what
record 0001 chose. It carries the three rules this tree holds itself to, it is
testable by the suite that already exists, and it adds no language, runtime or
dependency the tree does not already have. The one thing that is not Go is the
ruleset fetch, which is forced by the answer living on the platform rather than
in the checkout, and it is held to a single
gh apicall in the workflow.Nobody else has read this change. The approving review count on this board is
zero, so nothing refuses that, and the evidence above is what stands in place of
a reader rather than something added beside one.
What this does not do
It cannot judge behaviour. A job that reports the right name having verified
nothing passes this comparison exactly like one that did its work, and the proof
that a check bites is the fixture its own suite carries rather than the string it
reports under.
It reads the ruleset as the API answered on the day it ran, so a context added
or removed by hand is caught on the next pull request rather than at the moment
it changes. Between those two moments the gate and this comparison disagree and
nothing says so.
Both limits are written in the package comment where the comparison is defined,
because a green result here is otherwise read as the gate being intact right
now.
It does not make anything required. Nothing in this tree holds a merge today,
which is issue #26, and this check goes into the required set with the rest when
that lands.
For the two names a code-scanning upload reports under, this is a statement that
the name is expected rather than a statement about what the tree says. The
string those two report under is decided by the analysis rather than by a line
anybody can change here, so the rename this whole check is about is not caught
for them. That is written at the list, and the report says which of the two
claims a reader is holding.
The reader of the workflow files understands the shape the workflows here are
written in and refuses anything else rather than guessing. That bound is stated
at the reader, and a workflow written differently reddens this check instead of
silently declaring the wrong string.