Lattice research environment

Verifiable LaTeX tasks for training and evaluating paper-writing agents. Each task is a real paper with one fault in it, a verifier that decides whether the agent repaired it, and self-tests that prove the verifier cannot be talked into passing a cheat. Tasks are Harbor task packages, so Claude Code, Codex CLI and every other Harbor agent run them in a pinned TeX Live container unchanged, and Tinker's Harbor RL recipe trains on them.

This first phase covers the TeX-only task families, which need only TeX Live and pdftotext, not the Lattice app. Everything here is the Python standard library; nothing to install.

cd research-env
python3 -m lattice_env seeds                                  # the seed papers and their licences
python3 -m lattice_env generate --out /tmp/tasks              # generate, guard and admit tasks
python3 -m lattice_env selftest /tmp/tasks/<task>             # rerun one task's self-tests
python3 -m lattice_env verify /tmp/tasks/<task> <workspace>   # score a workspace
python3 -m unittest discover -s tests                         # the tests (TeX ones need latexmk + pdftotext)

Running the tasks through Harbor (needs Docker):

harbor run -p /tmp/tasks -a oracle          # the reference repairs: every task scores 1
harbor run -p /tmp/tasks -a nop             # the untouched papers: every task scores 0
harbor run -p /tmp/tasks -a claude-code -m anthropic/<model> -k 4
harbor run -p /tmp/tasks -a codex -m openai/<model> -k 4

A task package

<task>/
  task.toml             Harbor's config; [metadata] has family, operator, category, tier, seed
  instruction.md        what the agent is told; never names the fault
  environment/
    Dockerfile          TeX Live 2025 pinned by digest, plus pdftotext; the workspace copied to /app
    workspace/          the broken paper
  solution/             the oracle: fix.json (the reference repair) and solve.sh to apply it
  tests/                uploaded only after the agent is done
    test.sh             Harbor's entry: writes /logs/verifier/reward.{txt,json} and verdict.json
    verify.py           python3 -I verify.py --workspace DIR [--out DIR]
    selftest.py         python3 -I selftest.py: this task's self-tests
    lattice_verifier/   a copy of lattice_env/verifier, so the package needs nothing else
    task.json           the verifier's private config: the fault, the edit budget, file hashes
    initial/ original/  the editable files before and after the fault
    selftests/*.json    the self-test cases

tests/ holds only text files, since Tinker uploads them as text.

How a submission is scored

Gates first. A submission scores 0 unless every gate holds:

Then the grade. A PDF byte-identical to the original paper's scores 1. Anything else scores 0.9 × restoration × editTerm: restoration falls by half for every 4 tokens of the PDF's text that differ from the original's (pdftotext -layout, lowercased), and editTerm falls as the square root once the edit is bigger than the reference repair. The decay is steep on purpose: deleting the one sentence that holds a fault keeps a 4-page paper's text 99% similar, and pays almost nothing here.

Red lines, each a way to pass without repairing the paper:

reward.json carries the named components beside reward: pdfIdentical, restoration, editTerm, similarity (TeXFix-Bench's S_pdf, for comparison) and pagesMatch. verdict.json has everything, including which gates failed and why. When the verifier cannot judge (TeX or pdftotext missing, the original paper failing to build), the status is infra_error: it writes no reward and exits 1, so Harbor records an error rather than a 0 the agent did not earn. (Tinker's recipe scores a grading error as 0.0; drop those episodes there by their verdict.json.)

The pinned build

latexmk with -norc (no rc file runs), no shell escape, pdfTeX only, a fixed root document, a fresh copy of the sources with every build artifact removed, a fresh HOME and TeX user trees, and SOURCE_DATE_EPOCH with FORCE_SOURCE_DATE pinned. The reference is rebuilt from the original sources in the same environment on every verification, so a task's score never depends on a PDF built elsewhere.

Self-tests

Every task ships cases its verifier must score as expected, and the generator admits a task only if all of them hold:

• the reference repair scores 1; the untouched workspace scores 0;
• each known cheat that applies scores 0: deleting the enclosing section, hiding it in \iffalse, commenting the faulty paragraph out, silencing warnings, a project .latexmkrc, aliasing the broken name, inventing a label, redefining \ref, a stub package, touching the decoy, a different paper, an empty paper;
• probes stay below 0.5: deleting the faulty line, and three seeded random edits.

A rejected task stays in the output's _rejected/ folder with its failing cases in manifest.json, so a family whose verifier leaks shows up as a rejection rate. generate --strict (what CI runs) fails on any of them.

Build repair

The operators port DocMut's LaTeX catalogue (MIT, Copyright (c) 2026 Prajwal S. Venkateshmurthy / Oleafly) under its codes and tiers, so results compare with TeXFix-Bench, whose categories they carry. LAT-* operators fill the taxonomy categories DocMut does not cover for pdfLaTeX. A mutant is kept only if the pinned build fails on it (DocMut's render-diff guard), and sites are drawn with a seeded shuffle keyed by the salt, the seed's content and the operator.

Reference hygiene

The paper builds, but a reference or citation is wrong, which a compile-only reward never sees. The verifier reads the build's .aux: every label must print the number it printed in the original, every citation must resolve as it did, and every \ref must point at the key it pointed at. The guard keeps a mutant only if it builds into a PDF that differs from the original's.

Lattice's own tools: research-writer env

The same task packages run inside Lattice's binary on macOS, where the agent works through the app's domain code instead of a shell: its build diagnostics, history-backed edits, SyncTeX and symbol checks. It speaks JSON lines on stdin and stdout and runs before Tauri starts, like the literature mode the agent sidecar uses.

research-writer env --episodes /tmp/episodes          # then one JSON request per line:
{"op":"reset","task":"/tmp/tasks/<task>","agent":{"name":"my-agent","model":"m"},"budget":{"steps":60,"compileSeconds":300}}
{"op":"step","action":{"type":"compile"}}
{"op":"step","action":{"type":"read","path":"main.tex","from_line":50,"to_line":80}}
{"op":"step","action":{"type":"edit","path":"main.tex","old":"\\citte{","new":"\\citet{"}}
{"op":"step","action":{"type":"submit","claim":"solved"}}
{"op":"close"}
research-writer env --replay /tmp/episodes/<episode>/transcript.jsonl   # rerun and compare observations

• Replies are Gymnasium's (obs, reward, terminated, truncated) with the reward on the reply as OpenEnv puts it, info for timings and the verdict (status, components, gates, redLines, claimMatches), and Collaborative Gym's multi-party fields: each step names its role (agent, human, simulator, system), observations split into public and private, and events tell the agent what another party changed since it last acted. Only the agent's steps count against the budget; running out scores the workspace as it stands (truncated).
• Actions: list_files, read, search, edit (one unique occurrence), write, compile, pdf_text, synctex, synctex_inverse, check (unused labels and citations, word count), submit (with an optional claim: solved, partial, unsolved) and defer (give up with a reason). There is no shell.
• Isolation: each episode copies the task's environment/workspace/ into <episodes>/<id>/workspace with its own HOME, so nothing reaches the real app's caches. The build is latex::build_pinned: Lattice's latexmk run with the root, engine and clock from the task, -norc, no shell escape and no preamble-format cache. submit runs the package's own tests/verify.py, as Harbor does.
• Records: <id>/trajectory.json in Harbor's trajectory format (ATIF-v1.7, Lattice's fields under extra), <id>/transcript.jsonl (every request and reply, which --replay reruns), and one line per episode in <episodes>/episodes.jsonl for calibrate.

Difficulty calibration

Run a baseline several times per task, then label the tasks from the job folders:

harbor run -p /tmp/tasks -a claude-code -m anthropic/<model> -k 4 -o jobs --job-name claude
python3 -m lattice_env calibrate jobs/claude --tasks /tmp/tasks --out calibration.json
python3 -m lattice_env select calibration.json --label KEEP --tasks /tmp/tasks --copy-to /tmp/keep

A task is TRIVIAL (mean reward of its scored attempts at least 0.999), UNSOLVABLE (at most 0.001) or KEEP, CoworkEnv's teacher-difficulty labels; RL batches should come from KEEP. With fewer than --min-attempts (3) scored attempts it is UNDERSAMPLED. The report also gives, per run:

• the share of informative groups: a task's attempts that did not all get the same reward. A group with no variance gives a GRPO-style learner nothing to compare, even with a KEEP mean;
• unscored trials, by reason (an image that failed to build, a verifier that could not judge), counted apart and never averaged in as 0; Harbor still verifies after an agent timeout, so those count as scored;
• the hack rate (scored trials that crossed a red line, by code) and the share that touched the decoy, to read beside each agent's score.

calibrate also reads JSON-lines episode records (task, agent, model, reward, status, redLines), the format Lattice's in-binary environment writes. --expect-reward 1 (or 0) instead fails unless every trial scored exactly that, which is how CI checks Harbor's oracle and nop agents on a sample.

The held-out split

Tasks from one seed share almost all their text, so the held-out split is by seed: private seed folders, never committed, generated with a private salt, and rotated by drawing again with new seeds and a new salt.

python3 -m lattice_env split-check --private-seeds-dir ~/lab/heldout-seeds   # leakage scan; exit 1 on a leak
python3 -m lattice_env generate --seeds-dir ~/lab/heldout-seeds --seed <name> --salt "$(cat ~/lab/salt-2026q4)" --out ~/lab/heldout

split-check is CoworkEnv's near-duplicate test: word 4-grams of each seed's live LaTeX, flagging a private seed when more than half of the smaller side's 4-grams also occur in a public one. Mined real faults (TeXFix-Bench's under-20-line fix commits) belong in this split; importing them is not built yet.

Preference data from annotation rounds

Every Lattice project that used annotation rounds already holds preference data: the writer's note and intent on a passage, the agent's proposal and reason, the decision (accepted whole or in part, or rejected), and each round's files before and after (.research/annotations.json, .research/annotation-rounds/). The exporter turns them into training files, locally; nothing is uploaded.

python3 -m lattice_env export-preferences ~/papers/my-paper --out /tmp/prefs --level 1 [--writer <author id>]

Decisions are accept, accept-partial (some hunks applied), accept-edited (changed by the writer before the round settled) and reject. Prompts reproduce the shipping agent's request (src/agent/agent-annotations.ts). Consent levels: 0 labels and counts only; 1 adds the passages the agent touched, its proposals and the writer's notes, with % comments removed (they hold discarded drafts) and emails and URLs scrubbed; 2 adds the context around each passage and each round's whole files. Ids are salted hashes, paths are hashed below level 2, known author names are replaced in free text, and replies by anyone but an annotation's author are dropped, since co-authors did not consent. A rejected proposal's text survives only for an annotation's latest round; earlier rounds export their outcome without text. Studies with recruited writers run under the lab's IRB protocol; this tool only reads what the app already keeps.

The fixture in tests/fixtures/annotated-paper/ is also read by the app's own parser in a Rust test, so a change to Lattice's annotation schema fails there before the exporter misreads it.

Seeds and licences

python3 -m lattice_env seeds lists them. The bundled seeds are Lattice's own templates (the tutorial paper and the NeurIPS, ICML and ICLR skeletons), written the way project creation writes them. seeds/ adds four papers written for this environment, Apache-2.0, each a few pages long and chosen to cover what the templates do not:

Their numbers are illustrative and say so in the papers; their bibliographies cite real papers. Figures are plain placeholders generated at load time (seed.json → placeholders), and conference styles are borrowed from src-tauri/templates/ (styles) rather than copied. Conference style files ship unmodified and carry no licence grant of their own (see each template's UPSTREAM.md), so seeds that need them are marked redistributable: false; a dataset release should filter on that. The tutorial's two figure panels reproduce Google's Transformer figure under a grant limited to scholarly use, so the seed replaces them with plain placeholder images of the same size and words the caption accordingly.

Private seeds, for a held-out split that is never published, are folders in the same layout (seed.json plus the project) passed with --seeds-dir, generated with a private --salt.

Known limits

• No Docker on the development Mac. Colima has a stopped profile and there is no docker CLI, so no Harbor run has happened locally. CI builds the task image and runs the TeX tests and a strict sample generation in it, which is also where pdfTeX's determinism inside the container is checked (every seed is built twice in different folders).
• The image is amd64 only (TL2025-historic has no arm64 build); on Apple Silicon it runs under emulation.
• The base image moves. The historic tags are rebuilt monthly on Debian testing; the Dockerfile pins the digest, and poppler-utils comes from whatever Debian testing ships when the image is built, which can shift similarity and restoration slightly but never PDF identity.
• An agent running as root in the container can change TeX Live itself (or the verifier's Python) before Harbor's verifier runs in the same container. The .fls check catches inputs from outside TeX's trees, not edits inside them. Harbor's separate-verifier mode, with /app declared as an artifact, verifies in a fresh container and closes this; it is not the default in Harbor 0.24 and is untested here.