pill — Program Improvement Log Looker

pill audits a project using its output.log and prompts.log (from prr, story, or a previous pill run), then writes an improvement plan to pill-output.md and pill-summary.md. It does not run fixers, verify diffs, or commit — it is analysis-only.

---

Why pill exists

• prr and story produce logs that describe what the tool did and what the LLM saw. Those logs are evidence of behavior: failures, retries, model rotations, and user-visible output.
• WHY audit logs: Logs often reveal patterns that code review misses: repeated failures pointing to prompt or logic bugs, bail-outs that suggest architectural issues, or documentation gaps. Turning that evidence into an actionable improvement plan helps improve the project (and the tool itself when run on prr’s own repo).
• WHY triage when prr audited another repo: output.log mostly describes PRR’s work inside the clone (the PR’s codebase). The audit LLM often proposes improvements for that tree (src/, packages/, …) as well as for PRR (tools/prr/, shared/). Default (this repo layout): When targetDir contains tools/prr, pill post-filters so pill-output.md only records paths under tools/, shared/, tests/, docs/, generated/, .cursor/, .github/, or allowlisted root files — clone paths are omitted (see PILL_TOOL_REPO_SCOPE_FILTER). AGENTS.md / DEVELOPMENT.md (“Pill output triage”) and pill-output.md legend cover older runs and PILL_TOOL_REPO_SCOPE_FILTER=0.
• WHY analysis-only: Earlier designs included fix/verify/commit cycles. That duplicated prr’s loop and made pill heavy and stateful. Analysis-only keeps pill simple: one LLM audit, append-only markdown output, no git or runner integration. You (or another process) decide what to do with the plan.
• WHY two output files: pill-output.md holds the full improvement list (summary + per-item details) so implementers have one place to work from. pill-summary.md holds a short, dated log of runs (pitch + link to pill-output) so you can see history at a glance without opening the full instructions file.

---

How it works

1. Context assembly — Reads docs, source (with token budget), directory tree, and the target output.log and prompts.log. Log file names depend on logPrefix (see below). Large logs are summarized so the audit request stays within context and avoids 504 / FUNCTION_INVOCATION_TIMEOUT.
2. Audit LLM — Sends context to the configured audit model with a system prompt that asks for: a pitch (engaging 1–2 paragraph summary), a summary (technical overview), and improvements (file, description, rationale, severity, category).
3. Output — Appends one dated section to pill-output.md (full plan) and one entry to pill-summary.md (pitch + link). Uses .toLocaleString() for user-facing counts (no dependency on shared logger; see workspace rule).

Context assembly (large logs and 504 avoidance):

• When output.log is small (≤30k tokens and ≤100k chars), it is included in full. When it is large, we use head (first 400 lines) + story-read middle (chapter-by-chapter LLM summarization from shared/llm/story-read.ts) + tail (last 400 lines) + excerpt (high-signal lines: RESULTS SUMMARY, Model Performance, etc.). WHY: A full ~183k-char log caused 504; char and token thresholds trigger summarization earlier; head/tail preserve init and exit state.
• After assembly, the output log is capped at 28k chars by default. Override with PILL_OUTPUT_LOG_MAX_CHARS. WHY: Hard char cap + Vercel timeouts on large ElizaCloud audit bodies.
• When the log is large but no LLM client is available (e.g. dry-run), we send head + "[ … middle omitted (no summarization client) … ]" + tail + excerpt instead of raw. WHY: Sending raw in that path would still cause 504.
• prompts.log is included raw when under the same token threshold (each entry still capped at 12k chars); otherwise it is summarized via story-read. Each PROMPT/RESPONSE pair is truncated to ~18k chars before chapter grouping so a single 200k-char conflict prompt cannot create one giant story-read chapter. See shared/README.md for story-read and shared/utils/tokens.ts for truncation.

WHY logPrefix: When prr runs, logs are output.log / prompts.log. When story runs, the shared logger uses prefix: 'story' so files are story-output.log / story-prompts.log. Pill uses its own logger with hardcoded pill-output.log / pill-prompts.log. The prefix tells pill which log pair to read (prr vs story vs pill-on-pill).

WHY dynamic import in closeOutputLog: The shared logger’s closeOutputLog() runs pill via dynamic import('../tools/pill/orchestrator.js') so that the main process never loads pill at startup. That avoids a circular dependency: orchestrator used to import formatNumber from shared/logger.js, while logger imported the orchestrator in the close path. We broke the cycle by using n.toLocaleString() in the orchestrator and keeping the pill hook behind a dynamic import.

WHY errors propagate from runPillAnalysis when called from CLI: The pill CLI should show real failures (missing API key, LLM error, parse/write error). The shared logger’s hook wraps runPillAnalysis in try/catch and swallows errors so that shutdown always completes and pill remains optional. Only the CLI path gets thrown errors.

---

Usage

Standalone (pill CLI)

# From repo root (must contain or specify logs)
node dist/tools/pill/index.js <directory> [options]
# or after npm link
pill <directory> [options]

• <directory> — Directory that contains the log files and project to audit (e.g. . or ~/.prr if logs are there).
• --audit-model <model> — Model for the audit call (default: claude-opus-4-6).
• --output-only — Use only output.log (no prompts.log).
• --prompts-only — Use only prompts.log (no output.log).
• --dry-run — Run audit and show results; do not write pill-output.md or pill-summary.md.
• --instructions-out <path> — Override path for pill-output.md.
• -v, --verbose — Verbose logging (provider, model, token counts, plan preview).

Config (API keys, provider) is loaded from <directory>/.env and then ~/.pill/.env (target overrides home). Same env vars as prr/story (e.g. ELIZACLOUD_API_KEY, ANTHROPIC_API_KEY).

• PILL_CONTEXT_BUDGET_TOKENS (optional, 8000–128000) — Max context tokens for the assembled context (default 35000). Per-section caps scale with the budget. If you hit 504, try 20000 or lower.
• PILL_AUDIT_MAX_USER_CHARS (optional, 6000–80000) — Hard cap on user message size per audit HTTP request (single request or each chunk). If unset: default ceiling ~20k chars, ~12k for Opus-class / o3- / gpt-5 (excluding mini/nano) audit models (Vercel invocation timeout). Raise only if you use a direct API (not ElizaCloud) or a fast model.
• PILL_AUDIT_CHUNK_CONCURRENCY (optional, 1–16, default 4) — How many audit HTTP requests may run in parallel when context is split into chunks. Higher speeds large runs; use 1 to restore fully sequential behavior (e.g. strict rate limits).
• PILL_OUTPUT_LOG_MAX_CHARS (optional) — Hard cap on output-log chars in the audit payload (default 28000).
• PILL_TOOL_REPO_SCOPE_FILTER (optional) — 0 / false / off disables dropping clone-only paths; 1 / true forces the filter on. Unset: filter is on only when tools/prr exists under targetDir (typical prr monorepo). WHY: Keeps pill-output.md focused on improving this tool repo, not the PR under review.

Integrated (prr / story / split-exec / split-plan) — opt-in with --pill

When you run prr, story, split-exec, or split-plan with the --pill flag, the shared logger enables pill for that run. On normal exit (or Ctrl+C), closeOutputLog() runs pill on the logs that were just closed and prints the pitch and file paths to the real console (using the original console refs captured before patching). When --pill is not passed, pill does not run.

• WHY opt-in: Running pill on every run (especially for tools like split-exec that make no LLM calls) added cost and often produced empty or low-value output. Making pill explicit keeps default runs fast and lets users request analysis when they want it.
• WHY run on close: The logs are only complete and flushed after the tee streams are closed. Running pill before close would read incomplete or buffered content.
• WHY orig refs: The process console is patched to tee to the log file. Pill’s output (pitch, paths) should go to the user’s terminal, not into the log. We capture console.log/warn/error before patching and use those in the hook so the user sees pill’s message in the real console.
• WHY reset pillAnalysisEnabled at start of block: So the hook runs at most once even if something throws later; we set pillAnalysisEnabled = false before any await.
• WHY double-init guard in initOutputLog: If initOutputLog is called twice (e.g. in tests or re-entry), we must not overwrite the original console refs with the already-patched console. We only assign origLogRef / origWarnRef / origErrorRef when they are still null, so the first capture is preserved and the pill hook always has the real console.

---

Failure modes and debugging

When pill records no improvements, it returns a distinct reason so you can fix the cause (pill-output.md #3):

Where you see the reason: When pill is run from the CLI, it prints a single line (e.g. No improvements to record: No logs to analyze.). When pill runs from the hook (after prr/story/split-exec close their logs), the shared logger prints a short message to the console (e.g. [Pill] No logs to analyze (output/prompts log empty or missing for this prefix).) and appends a [Pill] No improvements to record (reason: …) line to the output log. Check pill-output.log (or the tool’s output log) for the exact reason string.

---

Output files

Numbers in user-facing strings (e.g. improvement counts) use .toLocaleString() for locale-aware formatting (e.g. comma thousands separators).

---

Log file names and prefix

Pill infers the pair from logPrefix in config (CLI passes it when invoked from the hook via currentLogPrefix). When auditing prr’s own repo, pill can also include pill-output.log in context when the primary logs are not pill’s own (pill-on-itself).

If pill says "every PROMPT/RESPONSE entry is empty" but your prompts.log has content: Pill reads both logs from the same directory (the one containing the output log). When pill runs from the prr hook, that directory is where prr wrote its logs. The "empty" conclusion means the prompts.log pill actually read had entry headers but no body text — either (1) a different run wrote that file (e.g. logging bug or different cwd), or (2) you are looking at a prompts.log in another directory. Run pill with the directory that contains the log pair you want (e.g. pill /path/to/dir), or run prr with --pill from the directory where you want both logs so the hook reads the same run's files.

---

Architecture (brief)

• cli.ts — Commander; parses <directory> and options; resolves directory to absolute path.
• config.ts — Loads .env, builds PillConfig. tryLoadPillConfig() returns null instead of throwing (used by the shared logger hook so missing config doesn’t break shutdown).
• context.ts — Assembles PillContext: docs, source, tree, output log, prompts digest; story-read for large logs (head+tail+summarized middle); char cap (PILL_OUTPUT_LOG_MAX_CHARS); pill-on-itself handling. Uses shared/utils/tokens.ts and shared/llm/story-read.ts.
• orchestrator.ts — runPillAnalysis(config): assemble context, call audit LLM, parse JSON, append to pill-output.md and pill-summary.md (or return paths in dry-run). No try/catch around the whole function so the CLI sees errors; the hook in logger catches and ignores.
• llm/prompts.ts — AUDIT_SYSTEM_PROMPT only (verify prompt was removed when the fixer was removed).
• types.ts — PillConfig, ImprovementPlan, Improvement, PillContext, etc.

---

See also

• Main README — prr, story, and pill overview.
• docs/README.md — Documentation index.
• shared/README.md — Shared tokens and story-read (WHYs, context caps).
• .cursor/rules/number-formatting.mdc — User-facing numbers use formatNumber from logger or n.toLocaleString().