From 8c015a3373bcec74c9fd367d6cc64449f0814d8b Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:42:36 +0200 Subject: [PATCH 1/6] fix(pipeline): retry the hand-off API calls that a GitHub brown-out killed On 2026-07-24 (~16:05-16:26 UTC) GitHub returned HTTP 502 on the workflow-dispatch endpoint and HTTP 504 on GraphQL. Four radar-basic PRs fell out of the pipeline and stayed there: #9742 was already ai-approved but impl-merge's condition check died on an unretried `gh pr view`, and #9744/#9749/#9751 were ai-rejected but their impl-repair dispatch never landed. impl-review's verdict step is the pipeline's only hand-off point - every downstream workflow starts from a call made there - so an unretried blip does not merely fail a step, it leaves a PR carrying a verdict label with nothing listening. - retry every API call in that step and in impl-merge's condition check 3x with linear backoff - dispatch impl-repair *before* adding the attempt label, so a label problem can never suppress the dispatch (losing the label costs one over-strict review; losing the dispatch costs the whole PR) - on dispatch exhaustion, drop `ai-rejected` on the way out: the combination ai-rejected + ai-attempt-N matches no watchdog case, while the label-less state is exactly Case 2, which re-dispatches the repair at the attempt number the remaining label still encodes - create `ai-attempt-N` on demand. `ai-attempt-4` never existed, so the 4th repair's label add always failed and `|| true` hid it (#7268/#7039 reached "Repair Attempt 4/4" carrying only ai-attempt-1..3). Every 4th review therefore re-applied the >= 90 attempt-0 bar, and the attempts-exhausted branch could never be reached - that branch's PR comment used a quoted heredoc and rendered a literal "$SCORE/100"; it now expands, with the markdown backticks escaped so they cannot become command substitution - impl-merge's merge retry re-reads state first: a merge that succeeded server-side but lost its response used to fail four more times with "already merged" and exit 1, skipping GCS promotion, the impl:{lib}:done label, closing the issue and the Postgres sync Also stop auto-updating Dependabot branches. That push is authored by github-actions[bot], which makes GitHub gate the resulting runs behind manual approval; because main advances every few minutes, each branch was re-updated long before anyone could approve. PR #9674 burned 174 runs in 22 h - 162 action_required against 4 green. They merge fine while behind (the main ruleset is not strict; the file's header claimed otherwise and is corrected), and /dependabot's playbook told operators to run that same harmful update-branch by hand. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/auto-update-pr-branches.yml | 51 +++++++-- .github/workflows/impl-merge.yml | 36 +++++- .github/workflows/impl-review.yml | 108 ++++++++++++++++-- CHANGELOG.md | 40 +++++++ agentic/commands/dependabot.md | 18 ++- 5 files changed, 232 insertions(+), 21 deletions(-) diff --git a/.github/workflows/auto-update-pr-branches.yml b/.github/workflows/auto-update-pr-branches.yml index 6327e35d1f..30119a22a4 100644 --- a/.github/workflows/auto-update-pr-branches.yml +++ b/.github/workflows/auto-update-pr-branches.yml @@ -2,12 +2,17 @@ name: "Auto-update PR branches" run-name: "Auto-update PR branches after push to ${{ github.ref_name }}" # When `main` advances, update every open PR that has auto-merge enabled -# and is BEHIND. Without this, the strict required-status-checks policy -# (`strict_required_status_checks_policy: true` on the `main` ruleset) -# blocks the auto-merge button forever — auto-merge fires only when checks -# are green AND the head is up-to-date with the base, but it does not -# update the branch on its own. This workflow closes that gap so polish / -# impl PRs squash-merge themselves end-to-end. +# and is BEHIND, so polish / impl PRs squash-merge themselves end-to-end +# without manual "Update branch" clicks. +# +# NOTE (verified 2026-07-24): the `main` ruleset currently sets +# `strict_required_status_checks_policy: false`, so an up-to-date head is +# NOT actually required to merge. This file used to claim the opposite. +# That means every update-branch call is optional work which also +# invalidates the green checks on the previous head and forces a full CI +# re-run. Retiring this workflow is worth evaluating — see #9675; it is +# kept for now because the ruleset could be tightened again, and because +# behaviour changes here affect every auto-merge PR in the repo. # # Why we don't use GitHub's native merge queue: this repository is # user-owned and merge_queue rules are not available on the account's @@ -62,9 +67,41 @@ jobs: --limit 200 \ --json number,title,headRefName,autoMergeRequest) + # Skip Dependabot branches. Updating them here is actively + # harmful: the update-branch merge commit is authored by + # github-actions[bot], and GitHub gates every workflow run + # triggered by that push behind manual approval + # ("action_required"). Because `main` advances every few minutes + # while the impl pipeline merges, the branch is re-updated long + # before anyone can approve — so CI never completes and + # auto-merge can never fire. PR #9674 accumulated 174 runs in + # 22 h that way: 162 action_required (github-actions[bot]) vs. + # only 4 green (dependabot[bot], from the original push). + # + # Leaving them alone is safe because the `main` ruleset is NOT + # strict (see the header) — a Dependabot PR merges while behind, + # so it never needs this update at all. It is explicitly NOT safe + # to assume Dependabot re-syncs them for us: Dependabot rebases on + # conflict / schedule / reopen, not on "merely behind", and it + # stops rebasing a PR entirely once a foreign commit is pushed to + # the branch. A branch already poisoned by such a commit stays + # stuck until someone comments `@dependabot recreate`. CANDIDATES=$(echo "$PRS" | jq -c ' - [ .[] | select(.autoMergeRequest != null) ] + [ .[] + | select(.autoMergeRequest != null) + | select(.headRefName | startswith("dependabot/") | not) + ] + ') + + SKIPPED=$(echo "$PRS" | jq ' + [ .[] + | select(.autoMergeRequest != null) + | select(.headRefName | startswith("dependabot/")) + ] | length ') + if [[ "$SKIPPED" -gt 0 ]]; then + echo "::notice::Skipping $SKIPPED Dependabot PR(s) — Dependabot rebases those itself (see comment above)" + fi COUNT=$(echo "$CANDIDATES" | jq 'length') echo "::notice::Found $COUNT open PR(s) with auto-merge enabled" diff --git a/.github/workflows/impl-merge.yml b/.github/workflows/impl-merge.yml index 5f450646b8..061d665449 100644 --- a/.github/workflows/impl-merge.yml +++ b/.github/workflows/impl-merge.yml @@ -47,7 +47,26 @@ jobs: run: | if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then PR_NUM="$INPUT_PR_NUMBER" - PR_DATA=$(gh pr view "$PR_NUM" --repo "$GH_REPO" --json headRefName,labels) + # Retry gh pr view — an unretried call here stranded an already + # ai-approved PR (#9742) when the GitHub GraphQL API returned + # HTTP 504 during the 2026-07-24 API brown-out. `bash -e` turned + # the blip into a permanently unmerged PR. Same 3x/linear-backoff + # shape as impl-review.yml's "Extract PR info". + PR_DATA="" + for attempt in 1 2 3; do + if gh pr view "$PR_NUM" --repo "$GH_REPO" --json headRefName,labels > /tmp/pr.json 2> /tmp/pr.err; then + PR_DATA=$(cat /tmp/pr.json) + break + fi + echo "::warning::gh pr view failed (attempt ${attempt}/3): $(cat /tmp/pr.err)" + if [ "$attempt" -lt 3 ]; then + sleep $((attempt * 5)) + fi + done + if [ -z "$PR_DATA" ]; then + echo "::error::gh pr view failed after 3 attempts for PR #${PR_NUM}" + exit 1 + fi BRANCH=$(echo "$PR_DATA" | jq -r '.headRefName') HAS_APPROVED=$(echo "$PR_DATA" | jq -r '[.labels[].name] | any(. == "ai-approved")') else @@ -224,6 +243,21 @@ jobs: for attempt in $(seq 1 $MAX_ATTEMPTS); do echo "::notice::Merge attempt $attempt/$MAX_ATTEMPTS" + # Re-read state first so a lost response is a no-op rather than a + # half-finished merge: `gh pr merge` can succeed server-side and + # still return 5xx, after which every later attempt fails with + # "already merged" and the step exits 1. Because all post-merge + # steps are gated on `should_run == 'true'` (implicit success()), + # that skipped GCS promotion, the impl:{lib}:done label, closing + # the issue and the Postgres sync — a merged PR with none of the + # bookkeeping, exactly the silent partial completion CLAUDE.md + # warns about for manually merged PRs. + STATE=$(gh pr view "$PR_NUM" --repo "$REPOSITORY" --json state -q .state 2>/dev/null || echo "") + if [ "$STATE" = "MERGED" ]; then + echo "::notice::PR #${PR_NUM} is already merged — continuing to post-merge steps" + exit 0 + fi + # Update branch before merge attempt gh pr update-branch "$PR_NUM" --repo "$REPOSITORY" 2>/dev/null || true sleep 2 diff --git a/.github/workflows/impl-review.yml b/.github/workflows/impl-review.yml index 27b9175894..95bb170a63 100644 --- a/.github/workflows/impl-review.yml +++ b/.github/workflows/impl-review.yml @@ -850,12 +850,58 @@ jobs: THRESHOLD=$((90 - ATTEMPT_COUNT * 10)) if [ "$THRESHOLD" -lt 50 ]; then THRESHOLD=50; fi - # Check if verdict label was already added by earlier step - CURRENT_LABELS=$(gh pr view "$PR_NUM" --json labels -q '.labels[].name' 2>/dev/null || echo "") + # This step is the pipeline's only hand-off point: every downstream + # workflow (merge, repair) starts from a call made right here. An + # unretried blip therefore does not just fail a step — it strands the + # PR forever with a verdict label and nothing listening. That is + # exactly what the 2026-07-24 GitHub API brown-out did to four + # radar-basic PRs (HTTP 502 on the dispatch endpoint, HTTP 504 on + # GraphQL). Every API call below is retried 3x with linear backoff. + # `::warning::` is line-oriented, so multi-line gh stderr would leak + # everything after the first newline into the raw log — flatten it. + gh_retry() { + local what="$1"; shift + local attempt err + for attempt in 1 2 3; do + if "$@" 2> /tmp/gh_retry.err; then + return 0 + fi + err=$(tr '\n' ' ' < /tmp/gh_retry.err | head -c 500) + echo "::warning::${what} failed (attempt ${attempt}/3): ${err}" + if [ "$attempt" -lt 3 ]; then + sleep $((attempt * 5)) + fi + done + echo "::error::${what} failed after 3 attempts for PR #${PR_NUM}" + return 1 + } + + # Check if verdict label was already added by earlier step. + # Track read success explicitly rather than inferring it from an + # empty result: a failed read and a genuinely label-less PR used to + # be indistinguishable, and both silently skipped the two branches + # below, dropping the PR out of the pipeline with no error at all. + CURRENT_LABELS="" + READ_OK=0 + for attempt in 1 2 3; do + if CURRENT_LABELS=$(gh pr view "$PR_NUM" --json labels -q '.labels[].name' 2> /tmp/labels.err); then + READ_OK=1 + break + fi + LERR=$(tr '\n' ' ' < /tmp/labels.err | head -c 500) + echo "::warning::gh pr view (labels) failed (attempt ${attempt}/3): ${LERR}" + if [ "$attempt" -lt 3 ]; then + sleep $((attempt * 5)) + fi + done + if [ "$READ_OK" -ne 1 ]; then + echo "::error::Could not read labels for PR #${PR_NUM} after 3 attempts — refusing to decide blind" + exit 1 + fi if echo "$CURRENT_LABELS" | grep -q "ai-approved"; then echo "::notice::ai-approved label already set (Score $SCORE >= $THRESHOLD), triggering merge" - gh workflow run impl-merge.yml -f pr_number="$PR_NUM" + gh_retry "impl-merge dispatch" gh workflow run impl-merge.yml -f pr_number="$PR_NUM" exit 0 fi @@ -863,19 +909,58 @@ jobs: # Still have repair attempts left? (Max 4 repairs = 5 reviews total) if [ "$ATTEMPT_COUNT" -lt 4 ]; then echo "::notice::ai-rejected label set (Score $SCORE < $THRESHOLD), triggering repair attempt $((ATTEMPT_COUNT + 1))" - gh pr edit "$PR_NUM" --add-label "ai-attempt-${ATTEMPT}" 2>/dev/null || true - gh workflow run impl-repair.yml \ + + # Dispatch BEFORE labelling, and never let the label gate the + # dispatch: impl-repair.yml is workflow_dispatch-only, so this + # call is the single thing that keeps the PR moving. Losing the + # attempt label costs one over-strict review; losing the dispatch + # costs the whole PR. + DISPATCHED=0 + if gh_retry "impl-repair dispatch" gh workflow run impl-repair.yml \ -f pr_number="$PR_NUM" \ -f specification_id="$SPEC_ID" \ -f library="$LIBRARY" \ -f attempt="$ATTEMPT" \ - -f model="$MODEL" + -f model="$MODEL"; then + DISPATCHED=1 + fi + + # `ai-attempt-4` was never created in this repo, so on the 4th + # repair the add fails deterministically no matter how often it + # is retried (PRs #7268/#7039 reached "Repair Attempt 4/4" + # carrying only ai-attempt-1..3 — the old `|| true` hid it). + gh label create "ai-attempt-${ATTEMPT}" --color "fbca04" \ + --description "Repair attempt ${ATTEMPT}" 2>/dev/null || true + # Non-fatal, but loud: without it the next review resets to the + # attempt-0 threshold (>= 90) and the PR re-fails a bar it has + # already cleared. PR #9744 lost it exactly this way on 2026-07-24. + gh_retry "ai-attempt-${ATTEMPT} label" gh pr edit "$PR_NUM" --add-label "ai-attempt-${ATTEMPT}" \ + || echo "::error::attempt label lost — the next review will apply the stricter attempt-0 threshold" + + if [ "$DISPATCHED" -ne 1 ]; then + # Hand the PR to the watchdog rather than leaving it dead. + # `ai-rejected` + `ai-attempt-N` matches NO watchdog case + # (Case 2 excludes any verdict label, Case 4 excludes PRs that + # already have an attempt label), so dropping the verdict puts + # it squarely in Case 2, which re-dispatches impl-repair at the + # attempt number the label still encodes. + gh pr edit "$PR_NUM" --remove-label "ai-rejected" 2>/dev/null || true + echo "::error::impl-repair dispatch failed after 3 attempts — verdict label dropped so watchdog Case 2 picks PR #${PR_NUM} up" + exit 1 + fi else # All 4 repair attempts exhausted echo "::notice::All 4 repair attempts exhausted (Score $SCORE < 50)" gh pr edit "$PR_NUM" --add-label "quality-poor" - gh pr comment "$PR_NUM" --body "$(cat <<'EOF' + # Unquoted delimiter so $SCORE/$LIBRARY/$REPOSITORY/$RUN_ID expand + # — with <<'EOF' this comment rendered a literal "$SCORE/100". + # The markdown code span's backticks MUST stay escaped: in an + # unquoted heredoc they would otherwise run as command + # substitution. Expanding the variables is safe even though + # $LIBRARY derives from a branch name, because bash does not + # re-evaluate substituted values. + gh pr comment "$PR_NUM" --body "$(cat < --auto --squash`. - Checks pending on an up-to-date branch → enable auto-merge anyway; it fires when green. - - **BEHIND branch**: update it via `gh api -X PUT repos/{owner}/{repo}/pulls//update-branch` - (the installed `gh` has no `pr update-branch` subcommand). **Gotcha:** branch updates pushed - with `GITHUB_TOKEN` (incl. `auto-update-pr-branches.yml`) do NOT trigger required checks — - if checks stay `expected`/missing after the update, disable auto-merge, then - `gh pr close && gh pr reopen ` to re-trigger them. + - **BEHIND branch**: leave it alone. BEHIND does not block the merge — the `main` ruleset is + not strict (`strict_required_status_checks_policy: false`), so auto-merge fires on a behind + branch. **Never** run `gh api -X PUT .../pulls//update-branch` on a Dependabot PR: the + resulting merge commit is authored by `github-actions[bot]`, GitHub then gates that head's + workflow runs behind manual approval (`action_required`, so the checks never report), and + Dependabot permanently stops rebasing a branch once a foreign commit lands on it. PR #9674 + burned 174 runs in 22 h this way. `auto-update-pr-branches.yml` skips Dependabot branches + for the same reason (#9675). + - **Already poisoned** (head commit authored by `github-actions[bot]`, checks stuck at + `action_required`): comment `@dependabot recreate` on the PR. That force-pushes a fresh + `dependabot[bot]`-authored head whose CI is not gated. Approving the pending run via + `gh api -X POST repos/{owner}/{repo}/actions/runs//approve` also works, but only holds + until something pushes to the branch again. - `mergeStateStatus` is computed async — after any update, poll it a few seconds until it stabilizes before deciding (UNKNOWN → BEHIND/CLEAN/BLOCKED). 3. A dep bump breaking tests/config (e.g. a major with changed exports): consult **Context7** From 4da745621832be06e87634f0fe644b400b438329 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:43:45 +0200 Subject: [PATCH 2/6] docs: point the changelog/comment refs at the real PR number Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/auto-update-pr-branches.yml | 2 +- CHANGELOG.md | 8 ++++---- agentic/commands/dependabot.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/auto-update-pr-branches.yml b/.github/workflows/auto-update-pr-branches.yml index 30119a22a4..4a01425fc4 100644 --- a/.github/workflows/auto-update-pr-branches.yml +++ b/.github/workflows/auto-update-pr-branches.yml @@ -10,7 +10,7 @@ run-name: "Auto-update PR branches after push to ${{ github.ref_name }}" # NOT actually required to merge. This file used to claim the opposite. # That means every update-branch call is optional work which also # invalidates the green checks on the previous head and forces a full CI -# re-run. Retiring this workflow is worth evaluating — see #9675; it is +# re-run. Retiring this workflow is worth evaluating — see #9772; it is # kept for now because the ruleset could be tightened again, and because # behaviour changes here affect every auto-merge PR in the repo. # diff --git a/CHANGELOG.md b/CHANGELOG.md index bacac8ea9e..ac8d8e3141 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -142,21 +142,21 @@ aggregate instead: an italic *Catalog* line at the end of the version section an never suppress it. If the dispatch still exhausts its retries, the step now drops the `ai-rejected` label on the way out: `ai-rejected` + `ai-attempt-N` matches no watchdog case, whereas the label-less state is exactly Case 2, which re-dispatches the repair at the right - attempt number (#9675). + attempt number (#9772). - **The 4th repair attempt was silently dead — `ai-attempt-4` never existed as a label** — the attempt label is what encodes the cascading review threshold (90 → 80 → 70 → 60 → 50), but the repo only ever had `ai-attempt-1..3`, so on the 4th repair the add failed every time and was swallowed by `|| true` (PRs #7268/#7039 reached "Repair Attempt 4/4" carrying only `ai-attempt-1..3`). Every 4th review therefore re-applied the ≥ 90 attempt-0 bar, and the "all attempts exhausted" branch — which closes the PR and removes the stale implementation — - could never be reached. The label is now created on demand (#9675). + could never be reached. The label is now created on demand (#9772). - **A merged implementation PR could end up with none of its bookkeeping** — `impl-merge`'s 5× merge retry re-invoked `gh pr merge` without re-reading state, so a merge that succeeded server-side but lost its HTTP response failed four more times with "already merged" and then exited 1. All post-merge steps are gated on `should_run == 'true'` (implicit `success()`), so GCS promotion, the `impl:{lib}:done` label, closing the issue and the Postgres sync were all skipped on an already-merged PR — the silent partial completion CLAUDE.md warns about. The - loop now checks for `MERGED` first and continues to the post-merge steps (#9675). + loop now checks for `MERGED` first and continues to the post-merge steps (#9772). - **Dependabot PRs could never merge while the plot pipeline was running** — `auto-update-pr-branches.yml` called `update-branch` on Dependabot PRs too, but that push is authored by `github-actions[bot]` and GitHub gates every workflow run it triggers behind @@ -168,7 +168,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an otherwise and has been corrected), and branches already poisoned by such a commit need a one-time `@dependabot recreate`, since Dependabot stops rebasing a branch once a foreign commit lands on it. `/dependabot`'s playbook told operators to run that same harmful - `update-branch` by hand and now says the opposite (#9675). + `update-branch` by hand and now says the opposite (#9772). - **anyplot.ai white-screen outage (2026-07-23, ~22:15–23:00 UTC): vite pinned back to 8.0.16** — the npm-minor Dependabot group (#9657) bumped vite 8.0.16 → 8.1.5, whose rolldown 1.1.5 bundler emits a broken `mui`/`@emotion` chunk under our `manualChunks` split; the diff --git a/agentic/commands/dependabot.md b/agentic/commands/dependabot.md index c47c7d3558..2edf464cdd 100644 --- a/agentic/commands/dependabot.md +++ b/agentic/commands/dependabot.md @@ -27,7 +27,7 @@ known GitHub quirks. Every gotcha below cost real session round-trips. workflow runs behind manual approval (`action_required`, so the checks never report), and Dependabot permanently stops rebasing a branch once a foreign commit lands on it. PR #9674 burned 174 runs in 22 h this way. `auto-update-pr-branches.yml` skips Dependabot branches - for the same reason (#9675). + for the same reason (#9772). - **Already poisoned** (head commit authored by `github-actions[bot]`, checks stuck at `action_required`): comment `@dependabot recreate` on the PR. That force-pushes a fresh `dependabot[bot]`-authored head whose CI is not gated. Approving the pending run via From 9419a1f3e514bf44cfebecc9f9cd7dc19f86c016 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:45:44 +0200 Subject: [PATCH 3/6] docs: correct the remedy for an already-updated Dependabot branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Approving the gated run once is enough — verified that the required contexts (Run Linting / Run Tests / Run Frontend Tests) do report as SUCCESS on a github-actions[bot]-authored head once the run is approved (#9674, commit 59545076). The head only kept getting re-gated because this workflow kept pushing to it; with that stopped, one approval sticks. `@dependabot recreate` is an alternative, not a requirement. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/auto-update-pr-branches.yml | 11 ++++++----- CHANGELOG.md | 8 ++++---- agentic/commands/dependabot.md | 10 ++++++---- 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/.github/workflows/auto-update-pr-branches.yml b/.github/workflows/auto-update-pr-branches.yml index 4a01425fc4..345494da84 100644 --- a/.github/workflows/auto-update-pr-branches.yml +++ b/.github/workflows/auto-update-pr-branches.yml @@ -80,12 +80,13 @@ jobs: # # Leaving them alone is safe because the `main` ruleset is NOT # strict (see the header) — a Dependabot PR merges while behind, - # so it never needs this update at all. It is explicitly NOT safe - # to assume Dependabot re-syncs them for us: Dependabot rebases on + # so it never needs this update at all. Do NOT justify this by + # assuming Dependabot re-syncs them for us: it rebases on # conflict / schedule / reopen, not on "merely behind", and it - # stops rebasing a PR entirely once a foreign commit is pushed to - # the branch. A branch already poisoned by such a commit stays - # stuck until someone comments `@dependabot recreate`. + # stops rebasing a branch entirely once a foreign commit lands on + # it. The point is that the head now stays put, so a branch this + # workflow already touched needs its gated run approved exactly + # once instead of being re-gated every few minutes. CANDIDATES=$(echo "$PRS" | jq -c ' [ .[] | select(.autoMergeRequest != null) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac8d8e3141..1177c6b133 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -165,10 +165,10 @@ aggregate instead: an italic *Catalog* line at the end of the version section an auto-merge never fired: PR #9674 accumulated 174 runs in 22 h — 162 `action_required` against only 4 green (the original `dependabot[bot]` push). Dependabot branches are now skipped. They merge fine while behind (the `main` ruleset is not strict — the file's header claimed - otherwise and has been corrected), and branches already poisoned by such a commit need a - one-time `@dependabot recreate`, since Dependabot stops rebasing a branch once a foreign - commit lands on it. `/dependabot`'s playbook told operators to run that same harmful - `update-branch` by hand and now says the opposite (#9772). + otherwise and has been corrected). An already-updated branch needs its gated run approved + once; that now sticks, because nothing pushes to the branch again. `/dependabot`'s playbook + told operators to run that same harmful `update-branch` by hand and now says the opposite + (#9772). - **anyplot.ai white-screen outage (2026-07-23, ~22:15–23:00 UTC): vite pinned back to 8.0.16** — the npm-minor Dependabot group (#9657) bumped vite 8.0.16 → 8.1.5, whose rolldown 1.1.5 bundler emits a broken `mui`/`@emotion` chunk under our `manualChunks` split; the diff --git a/agentic/commands/dependabot.md b/agentic/commands/dependabot.md index 2edf464cdd..41f3bf85f4 100644 --- a/agentic/commands/dependabot.md +++ b/agentic/commands/dependabot.md @@ -29,10 +29,12 @@ known GitHub quirks. Every gotcha below cost real session round-trips. burned 174 runs in 22 h this way. `auto-update-pr-branches.yml` skips Dependabot branches for the same reason (#9772). - **Already poisoned** (head commit authored by `github-actions[bot]`, checks stuck at - `action_required`): comment `@dependabot recreate` on the PR. That force-pushes a fresh - `dependabot[bot]`-authored head whose CI is not gated. Approving the pending run via - `gh api -X POST repos/{owner}/{repo}/actions/runs//approve` also works, but only holds - until something pushes to the branch again. + `action_required`): approve the gated runs once — + `gh api -X POST repos/{owner}/{repo}/actions/runs//approve`. Approved runs do report + the required contexts on a bot-authored head (verified on #9674), and since + `auto-update-pr-branches.yml` no longer touches these branches the head now stays put, so + one approval is enough. `@dependabot recreate` also works and additionally restores a + clean `dependabot[bot]` head, but it re-resolves the versions. - `mergeStateStatus` is computed async — after any update, poll it a few seconds until it stabilizes before deciding (UNKNOWN → BEHIND/CLEAN/BLOCKED). 3. A dep bump breaking tests/config (e.g. a major with changed exports): consult **Context7** From d60a508dc406aedbeef7c1a0392c17ade3c206dc Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:48:35 +0200 Subject: [PATCH 4/6] =?UTF-8?q?fix(pipeline):=20apply=20Copilot=20review?= =?UTF-8?q?=20=E2=80=94=20retry=20the=20watchdog=20hand-off=20and=20the=20?= =?UTF-8?q?state=20probe?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - impl-review: the `--remove-label ai-rejected` on the dispatch-failure path IS the hand-off to the watchdog, so it can't be an unretried `|| true` — the same brown-out would swallow it and leave the PR in the stranded state it exists to escape. Retried, and loud when it still fails. - impl-merge: retry the new state probe too. A blip there fell through to another `gh pr merge` on an already-merged PR, re-creating the merged-but-bookkeeping-skipped mode the probe was added to prevent. - impl-merge: flatten multi-line gh stderr before interpolating it into `::warning::`, matching impl-review. - auto-update-pr-branches: the ::notice:: still claimed "Dependabot rebases those itself", contradicting the corrected comment above it. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/auto-update-pr-branches.yml | 2 +- .github/workflows/impl-merge.yml | 25 +++++++++++++++++-- .github/workflows/impl-review.yml | 6 ++++- 3 files changed, 29 insertions(+), 4 deletions(-) diff --git a/.github/workflows/auto-update-pr-branches.yml b/.github/workflows/auto-update-pr-branches.yml index 345494da84..5cfd50baff 100644 --- a/.github/workflows/auto-update-pr-branches.yml +++ b/.github/workflows/auto-update-pr-branches.yml @@ -101,7 +101,7 @@ jobs: ] | length ') if [[ "$SKIPPED" -gt 0 ]]; then - echo "::notice::Skipping $SKIPPED Dependabot PR(s) — Dependabot rebases those itself (see comment above)" + echo "::notice::Skipping $SKIPPED Dependabot PR(s) — updating them gates their CI behind manual approval, and the non-strict ruleset means being behind does not block the merge (see comment above)" fi COUNT=$(echo "$CANDIDATES" | jq 'length') diff --git a/.github/workflows/impl-merge.yml b/.github/workflows/impl-merge.yml index 061d665449..c87a8792ea 100644 --- a/.github/workflows/impl-merge.yml +++ b/.github/workflows/impl-merge.yml @@ -58,7 +58,10 @@ jobs: PR_DATA=$(cat /tmp/pr.json) break fi - echo "::warning::gh pr view failed (attempt ${attempt}/3): $(cat /tmp/pr.err)" + # Flattened: `::warning::` is line-oriented, and gh's HTTP errors + # are routinely multi-line — the tail would spill into the raw log. + PERR=$(tr '\n' ' ' < /tmp/pr.err | head -c 500) + echo "::warning::gh pr view failed (attempt ${attempt}/3): ${PERR}" if [ "$attempt" -lt 3 ]; then sleep $((attempt * 5)) fi @@ -252,7 +255,25 @@ jobs: # the issue and the Postgres sync — a merged PR with none of the # bookkeeping, exactly the silent partial completion CLAUDE.md # warns about for manually merged PRs. - STATE=$(gh pr view "$PR_NUM" --repo "$REPOSITORY" --json state -q .state 2>/dev/null || echo "") + # The state read is itself retried: a blip here would fall through + # to another `gh pr merge` on an already-merged PR and re-create the + # very failure mode this check exists to prevent — under exactly the + # brown-out conditions that motivate it. + STATE="" + for probe in 1 2 3; do + if STATE=$(gh pr view "$PR_NUM" --repo "$REPOSITORY" --json state -q .state 2> /tmp/state.err); then + break + fi + SERR=$(tr '\n' ' ' < /tmp/state.err | head -c 500) + echo "::warning::gh pr view (state) failed (probe ${probe}/3): ${SERR}" + STATE="" + if [ "$probe" -lt 3 ]; then + sleep $((probe * 5)) + fi + done + if [ -z "$STATE" ]; then + echo "::warning::Could not read state for PR #${PR_NUM} — proceeding to merge attempt $attempt" + fi if [ "$STATE" = "MERGED" ]; then echo "::notice::PR #${PR_NUM} is already merged — continuing to post-merge steps" exit 0 diff --git a/.github/workflows/impl-review.yml b/.github/workflows/impl-review.yml index 95bb170a63..d8283311a9 100644 --- a/.github/workflows/impl-review.yml +++ b/.github/workflows/impl-review.yml @@ -944,7 +944,11 @@ jobs: # already have an attempt label), so dropping the verdict puts # it squarely in Case 2, which re-dispatches impl-repair at the # attempt number the label still encodes. - gh pr edit "$PR_NUM" --remove-label "ai-rejected" 2>/dev/null || true + # Retried too: this removal IS the hand-off to the watchdog, so + # letting the same brown-out swallow it would leave the PR in + # exactly the stranded state it is meant to escape. + gh_retry "ai-rejected removal" gh pr edit "$PR_NUM" --remove-label "ai-rejected" \ + || echo "::error::could not drop ai-rejected — PR #${PR_NUM} is stranded (ai-rejected + ai-attempt-N matches no watchdog case) and needs manual attention" echo "::error::impl-repair dispatch failed after 3 attempts — verdict label dropped so watchdog Case 2 picks PR #${PR_NUM} up" exit 1 fi From fb085ab3ef04fcbe27e453fa29e3b57916cc9c2a Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:53:12 +0200 Subject: [PATCH 5/6] fix(pipeline): truncate before flattening so the pipe can't SIGPIPE `tr ... | head -c 500` aborts the assignment under `set -o pipefail` once the input exceeds the 64 KB pipe buffer (verified: 200 KB input + pipefail -> abort; without pipefail -> rc 0). These steps run under plain `bash -e`, so it is not reachable today, but other steps in this repo do set `-eo pipefail` and gh's stderr is attacker-independent but unbounded. `head -c 500 | tr` is byte-identical output with no exposure either way. Applied to all four call sites, plus the pre-existing `$(cat /tmp/pr.err)` in "Extract PR info" which had the same multi-line annotation problem. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/impl-merge.yml | 4 ++-- .github/workflows/impl-review.yml | 7 ++++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/.github/workflows/impl-merge.yml b/.github/workflows/impl-merge.yml index c87a8792ea..7db97ee7b9 100644 --- a/.github/workflows/impl-merge.yml +++ b/.github/workflows/impl-merge.yml @@ -60,7 +60,7 @@ jobs: fi # Flattened: `::warning::` is line-oriented, and gh's HTTP errors # are routinely multi-line — the tail would spill into the raw log. - PERR=$(tr '\n' ' ' < /tmp/pr.err | head -c 500) + PERR=$(head -c 500 /tmp/pr.err | tr '\n' ' ') echo "::warning::gh pr view failed (attempt ${attempt}/3): ${PERR}" if [ "$attempt" -lt 3 ]; then sleep $((attempt * 5)) @@ -264,7 +264,7 @@ jobs: if STATE=$(gh pr view "$PR_NUM" --repo "$REPOSITORY" --json state -q .state 2> /tmp/state.err); then break fi - SERR=$(tr '\n' ' ' < /tmp/state.err | head -c 500) + SERR=$(head -c 500 /tmp/state.err | tr '\n' ' ') echo "::warning::gh pr view (state) failed (probe ${probe}/3): ${SERR}" STATE="" if [ "$probe" -lt 3 ]; then diff --git a/.github/workflows/impl-review.yml b/.github/workflows/impl-review.yml index d8283311a9..18f2c83d63 100644 --- a/.github/workflows/impl-review.yml +++ b/.github/workflows/impl-review.yml @@ -62,7 +62,8 @@ jobs: PR_DATA=$(cat /tmp/pr.json) break fi - echo "::warning::gh pr view failed (attempt ${attempt}/3): $(cat /tmp/pr.err)" + PERR=$(head -c 500 /tmp/pr.err | tr '\n' ' ') + echo "::warning::gh pr view failed (attempt ${attempt}/3): ${PERR}" if [ "$attempt" -lt 3 ]; then sleep $((attempt * 5)) fi @@ -866,7 +867,7 @@ jobs: if "$@" 2> /tmp/gh_retry.err; then return 0 fi - err=$(tr '\n' ' ' < /tmp/gh_retry.err | head -c 500) + err=$(head -c 500 /tmp/gh_retry.err | tr '\n' ' ') echo "::warning::${what} failed (attempt ${attempt}/3): ${err}" if [ "$attempt" -lt 3 ]; then sleep $((attempt * 5)) @@ -888,7 +889,7 @@ jobs: READ_OK=1 break fi - LERR=$(tr '\n' ' ' < /tmp/labels.err | head -c 500) + LERR=$(head -c 500 /tmp/labels.err | tr '\n' ' ') echo "::warning::gh pr view (labels) failed (attempt ${attempt}/3): ${LERR}" if [ "$attempt" -lt 3 ]; then sleep $((attempt * 5)) From d59219235d20505accb7da00d9936424938920b8 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Fri, 24 Jul 2026 22:57:18 +0200 Subject: [PATCH 6/6] fix(pipeline): treat "already merged" from gh pr merge as success MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The state probe narrows the lost-response window but doesn't close it: if the probe is blind for the same reason the merge response was lost, every retry reports "already merged" until the loop exits 1 — and the post-merge bookkeeping (GCS promotion, impl:{lib}:done, closing the issue, Postgres sync) is skipped on a PR that is in fact merged. gh saying the PR is already merged IS the success signal, so match it on stderr and continue. Genuine failures still retry and still exit 1, and the retry warning now carries the actual gh error instead of nothing. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/impl-merge.yml | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/.github/workflows/impl-merge.yml b/.github/workflows/impl-merge.yml index 7db97ee7b9..0ff40d12c0 100644 --- a/.github/workflows/impl-merge.yml +++ b/.github/workflows/impl-merge.yml @@ -296,14 +296,25 @@ jobs: --repo "$REPOSITORY" \ --squash \ --admin \ - --delete-branch; then + --delete-branch 2> /tmp/merge.err; then echo "::notice::Merge successful on attempt $attempt" exit 0 fi + # Second line of defence for the same lost-response case: if the + # state probe above is ALSO failing (same brown-out), the merge can + # already be done while every retry here reports "already merged" + # until the loop exits 1 and the post-merge bookkeeping is skipped. + # gh saying the PR is already merged IS the success signal. + MERR=$(head -c 500 /tmp/merge.err | tr '\n' ' ') + if grep -qiE 'already merged|pull request is (already )?(merged|closed)' /tmp/merge.err; then + echo "::notice::PR #${PR_NUM} reported as already merged — continuing to post-merge steps" + exit 0 + fi + if [ $attempt -lt $MAX_ATTEMPTS ]; then DELAY=$((attempt * 10)) - echo "::warning::Merge failed, retrying in ${DELAY}s..." + echo "::warning::Merge failed (${MERR}), retrying in ${DELAY}s..." sleep $DELAY fi done