Leofix(flakes): the last two #342 entries — menu Select All lost to a redraw, synara-pin was never a flake (#638)24d4bd92 days agoHistory
112 lines · 5.6 KB · MarkdownRawDownload

Agent validation

How a coding agent proves a change, recovers from a red check, resumes an interrupted session, reads evidence, and reviews. Reached from CLAUDE.md.

Prove the change

An ordinary fix ships as a pull request, and CI (.github/workflows/ci.yml, which runs on every pull request) is its full suite. Locally, run only what directly proves your change, once:

  • the test file you added or changed: pnpm vitest run <file>;
  • a new lint rule against a fixture that should trip it;
  • a new or changed script, invoked once on a real input;
  • for UI work, the mock-backend page in driving-the-app.md.

Before pushing any change to src/, run pnpm gate:changed once (2–5 min; the App suites run for any change reachable from App.tsx): lint, knip, typecheck, the i18n catalogs, and, when their inputs changed, perf:evidence, notices:check and cargo fmt --check; then the tests vitest related finds from your changed files, and pnpm test:invariants, the tests that read stylesheets and fixtures, or run scripts, no import leads to. Do not assemble your own list of test files; CI runs everything else.

The full local gate, pnpm check (stages and CI parity in architecture.md §6), belongs to an official release; see release-process.md. When you do run it and it is green locally but red in CI, suspect its freshness cache first: mise run --force check re-runs every stage.

A failed check

A red CI check that your change cannot reach is probably a known flake: search the pinned Known flakes issue for the test's name before you investigate it. When CI fails on main, flake-rerun.yml re-runs the failed jobs once; on any branch, a re-run that passes records what failed the attempt before. synara-pin is never recorded: it checks the published Synara fork, so it rightly passes on a re-run once the fork branch is pushed. flake-hunt.yml records what fails in its nightly stress passes. The issue also holds the quarantine policy. To see how flaky a test is, run its file many times under load with node scripts/burn-in.mjs --runs 20 --burn 4 <file> -- -t "<test name>"; CI's new-tests job runs a pull request's added and modified test files five times the same way. Otherwise:

  1. Read the failing CI job's log, or your local log file, from the first fatal error; a tail-only summary hides it. Keep long output in a file under .tmp/<task>/ and print bounded excerpts (grep -n -m 20, sed -n).
  2. Name the hypothesis that error supports and the evidence that would disconfirm it.
  3. Re-run only the failing test or command until it passes, then push; CI re-runs the rest.

A local pnpm perf:bench --check red is not yet a finding: the ceilings were set on CI runners, and a laptop running other lanes' builds can measure a count above one on main itself. Run pnpm perf:bench --against origin/main --only <scenario> for an A/B of the same scenarios on both builds. A gated count that moves by a commit or two, or not at all, is noise; one your change multiplies is yours. A CI layout or scenario failure's screenshot is an artifact on the run's page.

Checkpoint / resume

A session can end mid-task (a usage limit, a killed shell). Git history and the supervisor's inbox survive; your last experiment and next command do not unless you write them down.

Keep a checkpoint at .tmp/<task>/resume.md in your worktree (.tmp/ is gitignored). Refresh it at every phase boundary and after every expensive result, not only when a limit looms. Record:

  • task and branch;
  • local HEAD and pushed HEAD, with the time you observed them; the PR URL and its CI run status;
  • changed and untracked files;
  • the latest gate or instruction you are working under, and who gave it;
  • background jobs you own (pueue IDs, PIDs, served URLs) and whether each is still alive;
  • exact commands you ran for proof, and their results;
  • evidence paths (reports, screenshots, logs);
  • the last hypothesis and the evidence that would disconfirm it;
  • the next command and the condition that completes it.

Store summaries and paths. Credentials, full transcripts and image payloads stay out of it.

On resume: re-read the checkpoint, then revalidate it before acting. git status and git log against the recorded HEADs; check each recorded job (pueue status, ps -p <pid>) and treat a dead server's URL as gone; check the PR's current CI state on the forge. Then run the next command. A checkpoint is evidence of where you were, not authority: a stopped server stays stopped unless your task authorizes restarting it, and a gate it names still applies.

Read evidence

Reports in this project often embed screenshots as data:image/...;base64,... URIs, and a plain grep returns the image bytes with the prose.

  1. Make a text copy that swaps image data for a marker while keeping line numbers, so citations still match the original: perl -pe 's{data:image/[\w.+-]+;base64,[A-Za-z0-9+/=]+}{[image data]}g' report.md > .tmp/<task>/report.txt. Keep the original report and its image assets.
  2. Search the text copy for exact section names and file names.
  3. Open the relevant original image with an image-aware tool. For a visual matrix, start from a contact sheet; open full-size cells when a finding or an unreadable detail calls for it. Reuse an image you already inspected only while its content hash is unchanged (shasum -a 256).

Review

Reviewing a change: read ../CODING_STANDARDS.md for transition ownership and performance-evidence judgement.