Unified · Split
| @@ -0,0 +1,105 @@ | ||
| 1 | +# Executor apps — self-hosted build (DESIGN.md) | |
| 2 | + | |
| 3 | +Status: in progress. Durable record of the architecture, seam signatures, | |
| 4 | +package layout, key decisions, verification commands, and known gaps for the | |
| 5 | +executor apps subsystem built into the self-hosted deployment. | |
| 6 | + | |
| 7 | +Note: the tracked design-system doc is `design.md` (lowercase). This uppercase | |
| 8 | +`DESIGN.md` is the apps-subsystem architecture record required by the build | |
| 9 | +brief. | |
| 10 | + | |
| 11 | +## What this is | |
| 12 | + | |
| 13 | +User-authored, git-backed, published units — **custom tools**, **durable | |
| 14 | +workflows**, **UI views**, **skills** — published into a per-scope store and | |
| 15 | +served/executed by the self-hosted platform. Identity is the file path | |
| 16 | +(`tools/issues-sync.ts` IS the tool `issues-sync`). Publish is the compiler | |
| 17 | +(FDI): catalog entries, schedules, ui resources and the skills index are | |
| 18 | +projections of a versioned descriptor extracted from source at publish. | |
| 19 | + | |
| 20 | +The subsystem lives in one package, `@executor-js/plugin-apps`, wired into the | |
| 21 | +self-host app the same way every other plugin is (a source plugin in | |
| 22 | +`executor.config.ts`, HTTP routes as an extension, MCP tools/resources through | |
| 23 | +the MCP build hook). Everything substrate-specific sits behind a **seam** with | |
| 24 | +a substrate-neutral interface and a conformance suite that runs against the | |
| 25 | +interface, so future Cloudflare backings drop in without touching the | |
| 26 | +subsystem's logic. | |
| 27 | + | |
| 28 | +## The five seams | |
| 29 | + | |
| 30 | +Each seam is a substrate-neutral interface. Self-hosted backings are built now; | |
| 31 | +cloud backings are future. Everything crossing `ToolSandbox` is serializable | |
| 32 | +(the cloud version is RPC). | |
| 33 | + | |
| 34 | +| Seam | Self-hosted backing (built) | Cloud backing (future) | | |
| 35 | +|---|---|---| | |
| 36 | +| `ArtifactStore` | bare git repo per scope on disk (git CLI subprocess); `SnapshotId` = commit hash | Cloudflare Artifacts | | |
| 37 | +| `ScopeDb` | one libSQL/SQLite file per scope + per-table version counters | DO facets | | |
| 38 | +| `ToolSandbox` | QuickJS kernel (collect + invoke via the `SandboxToolInvoker` bridge) | Worker Loaders | | |
| 39 | +| `WorkflowRunner` | SQLite event-sourced journal replay runner + in-process scheduler | CF Workflows + dynamic-workflows | | |
| 40 | +| `LiveChannel` | in-process emitter + SSE | DO/facet socket owner | | |
| 41 | + | |
| 42 | +See `src/seams/*.ts` for the exact interfaces and `src/seams/*.conformance.ts` | |
| 43 | +for the suite each backing must pass. | |
| 44 | + | |
| 45 | +## Decisions (and why) | |
| 46 | + | |
| 47 | +- **Sandbox = QuickJS** (`packages/kernel/runtime-quickjs`). Its | |
| 48 | + `CodeExecutor.execute(code, toolInvoker)` already gives the serializable | |
| 49 | + handle bridge: `SandboxToolInvoker.invoke({path, args})` crosses as JSON, | |
| 50 | + `tools.<...>()` is a Proxy in the sandbox, `fetch` is disabled, there is a | |
| 51 | + deadline interrupt and a memory cap. secure-exec evaluated and rejected | |
| 52 | + (pre-1.0, per-arch native sidecar, flat string bridge fighting the Proxy | |
| 53 | + pattern); the Deno subprocess kernel is the documented harder-isolation | |
| 54 | + escalation behind the same seam. Because QuickJS evaluates a *string*, the | |
| 55 | + collect/invoke wrappers own the module shape: the published bundle is a | |
| 56 | + self-executing script that either records `define*()` descriptors (collect) | |
| 57 | + or calls one handler with injected clients (invoke). | |
| 58 | +- **Storage via host facades, not new tables.** Executor plugins deliberately | |
| 59 | + do not contribute FumaDB tables (`collectTables()` is fixed and | |
| 60 | + plugin-independent). App metadata (descriptors, snapshot pointers, schedules, | |
| 61 | + workflow journal, ui metadata) lives in `pluginStorage` collections; large | |
| 62 | + opaque blobs (compiled bundles, snapshot manifests) live in the `blobs` | |
| 63 | + facade, content-addressed by SHA-256. | |
| 64 | +- **ScopeDb is separate from the executor DB.** App *data* (the `issues` table | |
| 65 | + authors read/write) is one libSQL file per scope, independent of the | |
| 66 | + executor's own DB. Per-table version counters live alongside it and drive | |
| 67 | + `LiveChannel`. | |
| 68 | +- **Apps are a plugin source.** A published app maps to one executor | |
| 69 | + *integration* per scope (`apps`); a *connection* to it makes the published | |
| 70 | + tools catalog citizens through `resolveTools`/`invokeTool`, so | |
| 71 | + policy/approval/audit/toolkits/tools.list all apply unchanged. | |
| 72 | +- **No @effect/workflow.** The local runner is a purpose-built SQLite | |
| 73 | + event-sourced journal modeled on vercel/workflow's `World` Storage contract | |
| 74 | + (append-only events, materialized run/step views, replay-on-resume). | |
| 75 | + | |
| 76 | +## Package layout | |
| 77 | + | |
| 78 | +``` | |
| 79 | +packages/plugins/apps/ | |
| 80 | + src/ | |
| 81 | + seams/ ArtifactStore, ScopeDb, ToolSandbox, WorkflowRunner, LiveChannel | |
| 82 | + + one <seam>.conformance.ts per seam | |
| 83 | + pipeline/ discover -> bundle -> collect -> project (publish = the compiler) | |
| 84 | + plugin/ the apps source plugin (resolveTools/invokeTool), descriptor store | |
| 85 | + workflow/ journal runner + scheduler (over WorkflowRunner seam) | |
| 86 | + http/ publish + invoke + ui-bundle + SSE routes (HttpApiGroup + handlers) | |
| 87 | + mcp/ publish tool, ui:// resources, skills list/read over MCP | |
| 88 | + ui/ widget shell (browser render, tools/query bridge, SSE refetch) | |
| 89 | + testing/ test harness helpers + the daily-brief fixture set + e2e | |
| 90 | +``` | |
| 91 | + | |
| 92 | +## Verification gates — exact commands | |
| 93 | + | |
| 94 | +Run from the workspace root | |
| 95 | +(`usefulsoftwareco/.rifts/executor/apps-build-b`): | |
| 96 | + | |
| 97 | +- Typecheck (repo root): `bun run typecheck` | |
| 98 | +- Package tests (conformance + integration + e2e): | |
| 99 | + `bun run --filter='@executor-js/plugin-apps' test` | |
| 100 | + | |
| 101 | +Outputs pasted in the final report. | |
| 102 | + | |
| 103 | +## Known gaps | |
| 104 | + | |
| 105 | +Tracked here as the build proceeds; honest list in the final report. | |
| @@ -781,6 +781,37 @@ · 780 unmodified lines | ||
| 781 | 781 | "effect": "catalog:", |
| 782 | 782 | }, |
| 783 | 783 | }, |
| 784 | + "packages/plugins/apps": { | |
| 785 | + "name": "@executor-js/plugin-apps", | |
| 786 | + "version": "0.1.0", | |
| 787 | + "dependencies": { | |
| 788 | + "@executor-js/codemode-core": "workspace:*", | |
| 789 | + "@executor-js/config": "workspace:*", | |
| 790 | + "@executor-js/runtime-quickjs": "workspace:*", | |
| 791 | + "@executor-js/sdk": "workspace:*", | |
| 792 | + "@libsql/client": "catalog:", | |
| 793 | + "@standard-schema/spec": "^1.0.0", | |
| 794 | + "esbuild": "^0.27.3", | |
| 795 | + "zod": "4.3.6", | |
| 796 | + }, | |
| 797 | + "devDependencies": { | |
| 798 | + "@effect/vitest": "catalog:", | |
| 799 | + "@executor-js/api": "workspace:*", | |
| 800 | + "@modelcontextprotocol/sdk": "^1.29.0", | |
| 801 | + "@types/node": "catalog:", | |
| 802 | + "bun-types": "catalog:", | |
| 803 | + "effect": "catalog:", | |
| 804 | + "tsup": "catalog:", | |
| 805 | + "vitest": "catalog:", | |
| 806 | + }, | |
| 807 | + "peerDependencies": { | |
| 808 | + "@executor-js/api": "workspace:*", | |
| 809 | + "effect": "catalog:", | |
| 810 | + }, | |
| 811 | + "optionalPeers": [ | |
| 812 | + "@executor-js/api", | |
| 813 | + ], | |
| 814 | + }, | |
| 784 | 815 | "packages/plugins/desktop-settings": { |
| 785 | 816 | "name": "@executor-js/plugin-desktop-settings", |
| 786 | 817 | "version": "1.5.28", |
| @@ -1215,9 +1246,9 @@ · 428 unmodified lines | ||
| 1215 | 1246 | }, |
| 1216 | 1247 | }, |
| 1217 | 1248 | "patchedDependencies": { |
| 1218 | − "@1password/sdk-core@0.4.1-beta.1": "patches/@1password%2Fsdk-core@0.4.1-beta.1.patch", | |
| 1219 | 1249 | "@electric-sql/pglite-socket@0.1.4": "patches/@electric-sql%2Fpglite-socket@0.1.4.patch", |
| 1220 | 1250 | "libsql@0.5.29": "patches/libsql@0.5.29.patch", |
| 1251 | + "@1password/sdk-core@0.4.1-beta.1": "patches/@1password%2Fsdk-core@0.4.1-beta.1.patch", | |
| 1221 | 1252 | "agents@0.17.3": "patches/agents@0.17.3.patch", |
| 1222 | 1253 | "postgres@3.4.9": "patches/postgres@3.4.9.patch", |
| 1223 | 1254 | }, |
| @@ -1776,6 +1807,8 @@ · 552 unmodified lines | ||
| 1776 | 1807 | |
| 1777 | 1808 | "@executor-js/motel": ["@executor-js/motel@0.2.5-executor.1", "", { "dependencies": { "@effect/atom-react": "^4.0.0-beta.49", "@effect/opentelemetry": "^4.0.0-beta.49", "@effect/platform-bun": "^4.0.0-beta.50", "@opentelemetry/api": "^1.9.0", "@opentelemetry/exporter-logs-otlp-http": "^0.214.0", "@opentelemetry/exporter-trace-otlp-http": "^0.211.0", "@opentelemetry/sdk-logs": "^0.214.0", "@opentelemetry/sdk-node": "^0.214.0", "@opentelemetry/sdk-trace-base": "^2.5.0", "@opentelemetry/sdk-trace-node": "^2.6.1", "@opentui/core": "^0.1.99", "@opentui/react": "^0.1.99", "effect": "^4.0.0-beta.49", "react": "^19.2.5", "scheduler": "^0.27.0" }, "bin": { "motel": "src/motel.ts", "motel-mcp": "src/mcp.ts" } }, "sha512-fLGJ5FIIBYd+4L2OurpYFjWzvcCbACvmdNofh6THXQJE1Gku93EKURtgn27/XddyM9SKGST6/znTRcmwKiYdXw=="], |
| 1778 | 1809 | |
| 1810 | + "@executor-js/plugin-apps": ["@executor-js/plugin-apps@workspace:packages/plugins/apps"], | |
| 1811 | + | |
| 1779 | 1812 | "@executor-js/plugin-desktop-settings": ["@executor-js/plugin-desktop-settings@workspace:packages/plugins/desktop-settings"], |
| 1780 | 1813 | |
| 1781 | 1814 | "@executor-js/plugin-encrypted-secrets": ["@executor-js/plugin-encrypted-secrets@workspace:packages/plugins/encrypted-secrets"], |
| @@ -0,0 +1,60 @@ | ||
| 1 | +{ | |
| 2 | + "name": "@executor-js/plugin-apps", | |
| 3 | + "version": "0.1.0", | |
| 4 | + "homepage": "https://github.com/UsefulSoftwareCo/executor/tree/main/packages/plugins/apps", | |
| 5 | + "bugs": { | |
| 6 | + "url": "https://github.com/UsefulSoftwareCo/executor/issues" | |
| 7 | + }, | |
| 8 | + "license": "MIT", | |
| 9 | + "repository": { | |
| 10 | + "type": "git", | |
| 11 | + "url": "git+https://github.com/UsefulSoftwareCo/executor.git", | |
| 12 | + "directory": "packages/plugins/apps" | |
| 13 | + }, | |
| 14 | + "files": [ | |
| 15 | + "dist" | |
| 16 | + ], | |
| 17 | + "type": "module", | |
| 18 | + "exports": { | |
| 19 | + ".": "./src/index.ts", | |
| 20 | + "./api": "./src/api.ts", | |
| 21 | + "./seams": "./src/seams/index.ts", | |
| 22 | + "./testing": "./src/testing/index.ts" | |
| 23 | + }, | |
| 24 | + "scripts": { | |
| 25 | + "build": "tsup && (tsc --declaration --emitDeclarationOnly --outDir dist --rootDir src || true)", | |
| 26 | + "typecheck": "tsgo --noEmit", | |
| 27 | + "test": "vitest run", | |
| 28 | + "test:watch": "vitest", | |
| 29 | + "typecheck:slow": "bunx tsc --noEmit -p tsconfig.json" | |
| 30 | + }, | |
| 31 | + "dependencies": { | |
| 32 | + "@executor-js/codemode-core": "workspace:*", | |
| 33 | + "@executor-js/config": "workspace:*", | |
| 34 | + "@executor-js/runtime-quickjs": "workspace:*", | |
| 35 | + "@executor-js/sdk": "workspace:*", | |
| 36 | + "@libsql/client": "catalog:", | |
| 37 | + "@standard-schema/spec": "^1.0.0", | |
| 38 | + "esbuild": "^0.27.3", | |
| 39 | + "zod": "4.3.6" | |
| 40 | + }, | |
| 41 | + "devDependencies": { | |
| 42 | + "@effect/vitest": "catalog:", | |
| 43 | + "@executor-js/api": "workspace:*", | |
| 44 | + "@modelcontextprotocol/sdk": "^1.29.0", | |
| 45 | + "@types/node": "catalog:", | |
| 46 | + "bun-types": "catalog:", | |
| 47 | + "effect": "catalog:", | |
| 48 | + "tsup": "catalog:", | |
| 49 | + "vitest": "catalog:" | |
| 50 | + }, | |
| 51 | + "peerDependencies": { | |
| 52 | + "@executor-js/api": "workspace:*", | |
| 53 | + "effect": "catalog:" | |
| 54 | + }, | |
| 55 | + "peerDependenciesMeta": { | |
| 56 | + "@executor-js/api": { | |
| 57 | + "optional": true | |
| 58 | + } | |
| 59 | + } | |
| 60 | +} | |
| @@ -0,0 +1,284 @@ | ||
| 1 | +import { execFile } from "node:child_process"; | |
| 2 | +import { mkdir } from "node:fs/promises"; | |
| 3 | +import { join } from "node:path"; | |
| 4 | + | |
| 5 | +import { Effect } from "effect"; | |
| 6 | + | |
| 7 | +import { | |
| 8 | + ArtifactStoreError, | |
| 9 | + asSnapshotId, | |
| 10 | + type ArtifactStore, | |
| 11 | + type FileSet, | |
| 12 | + type ScopeArtifactStore, | |
| 13 | + type SnapshotId, | |
| 14 | + type SnapshotMeta, | |
| 15 | +} from "../seams/artifact-store"; | |
| 16 | + | |
| 17 | +// --------------------------------------------------------------------------- | |
| 18 | +// Git-backed ArtifactStore (self-hosted). One bare git repo per scope under | |
| 19 | +// `<root>/<scope>.git`. Snapshots are commits, written via git plumbing | |
| 20 | +// (hash-object -> update-index -> write-tree -> commit-tree) so no working tree | |
| 21 | +// is ever checked out — the runtime only reads committed snapshots. The commit | |
| 22 | +// hash IS the SnapshotId; git guarantees immutability (content-addressed). | |
| 23 | +// --------------------------------------------------------------------------- | |
| 24 | + | |
| 25 | +const BRANCH = "refs/heads/main"; | |
| 26 | + | |
| 27 | +const run = ( | |
| 28 | + cwd: string, | |
| 29 | + args: readonly string[], | |
| 30 | + input?: string | Buffer, | |
| 31 | +): Effect.Effect<string, ArtifactStoreError> => | |
| 32 | + Effect.async<string, ArtifactStoreError>((resume) => { | |
| 33 | + const child = execFile( | |
| 34 | + "git", | |
| 35 | + args, | |
| 36 | + { cwd, maxBuffer: 256 * 1024 * 1024, encoding: "buffer" }, | |
| 37 | + (error, stdout, stderr) => { | |
| 38 | + if (error) { | |
| 39 | + resume( | |
| 40 | + Effect.fail( | |
| 41 | + new ArtifactStoreError({ | |
| 42 | + message: `git ${args[0]} failed: ${(stderr as Buffer)?.toString() || error.message}`, | |
| 43 | + cause: error, | |
| 44 | + }), | |
| 45 | + ), | |
| 46 | + ); | |
| 47 | + return; | |
| 48 | + } | |
| 49 | + resume(Effect.succeed((stdout as Buffer).toString("utf8"))); | |
| 50 | + }, | |
| 51 | + ); | |
| 52 | + if (input !== undefined) { | |
| 53 | + child.stdin?.write(input); | |
| 54 | + child.stdin?.end(); | |
| 55 | + } | |
| 56 | + }); | |
| 57 | + | |
| 58 | +const sanitizeScope = (scope: string): string => { | |
| 59 | + if (!/^[a-zA-Z0-9._-]+$/.test(scope)) { | |
| 60 | + // Scopes are internal identifiers; refuse anything that could escape a path. | |
| 61 | + throw new ArtifactStoreError({ message: `invalid scope key: ${scope}` }); | |
| 62 | + } | |
| 63 | + return scope; | |
| 64 | +}; | |
| 65 | + | |
| 66 | +const makeScopeStore = (repoDir: string): ScopeArtifactStore => { | |
| 67 | + // Environment forcing a deterministic author/committer so a snapshot's hash | |
| 68 | + // depends only on its content + parent, not on wall-clock/identity drift. | |
| 69 | + const commitEnv = (message: string) => | |
| 70 | + Effect.gen(function* () { | |
| 71 | + const parent = yield* headCommit(); | |
| 72 | + // Build a tree from the provided file set held in a bare index file. | |
| 73 | + return { parent, message }; | |
| 74 | + }); | |
| 75 | + | |
| 76 | + const headCommit = (): Effect.Effect<string | null, ArtifactStoreError> => | |
| 77 | + run(repoDir, ["rev-parse", "--verify", "--quiet", BRANCH]).pipe( | |
| 78 | + Effect.map((out) => out.trim() || null), | |
| 79 | + Effect.catchAll(() => Effect.succeed(null)), | |
| 80 | + ); | |
| 81 | + | |
| 82 | + const readMeta = (id: string): Effect.Effect<SnapshotMeta, ArtifactStoreError> => | |
| 83 | + run(repoDir, ["show", "-s", "--format=%H%n%ct%n%s", id]).pipe( | |
| 84 | + Effect.map((out) => { | |
| 85 | + const [hash, committed, ...subjectParts] = out.split("\n"); | |
| 86 | + return { | |
| 87 | + id: asSnapshotId(hash.trim()), | |
| 88 | + committedAt: Number(committed.trim()) * 1000, | |
| 89 | + message: subjectParts.join("\n").trim(), | |
| 90 | + } satisfies SnapshotMeta; | |
| 91 | + }), | |
| 92 | + ); | |
| 93 | + | |
| 94 | + return { | |
| 95 | + commit: (files: FileSet, message: string) => | |
| 96 | + Effect.gen(function* () { | |
| 97 | + const { parent } = yield* commitEnv(message); | |
| 98 | + // Use a throwaway index so we never touch a working tree. | |
| 99 | + const indexFile = join( | |
| 100 | + repoDir, | |
| 101 | + `commit-index-${Date.now()}-${Math.random().toString(36).slice(2)}`, | |
| 102 | + ); | |
| 103 | + const withIndex = (args: readonly string[], input?: string | Buffer) => | |
| 104 | + Effect.async<string, ArtifactStoreError>((resume) => { | |
| 105 | + const child = execFile( | |
| 106 | + "git", | |
| 107 | + args, | |
| 108 | + { | |
| 109 | + cwd: repoDir, | |
| 110 | + maxBuffer: 256 * 1024 * 1024, | |
| 111 | + encoding: "buffer", | |
| 112 | + env: { ...process.env, GIT_INDEX_FILE: indexFile }, | |
| 113 | + }, | |
| 114 | + (error, stdout, stderr) => { | |
| 115 | + if (error) { | |
| 116 | + resume( | |
| 117 | + Effect.fail( | |
| 118 | + new ArtifactStoreError({ | |
| 119 | + message: `git ${args[0]} failed: ${(stderr as Buffer)?.toString() || error.message}`, | |
| 120 | + cause: error, | |
| 121 | + }), | |
| 122 | + ), | |
| 123 | + ); | |
| 124 | + return; | |
| 125 | + } | |
| 126 | + resume(Effect.succeed((stdout as Buffer).toString("utf8"))); | |
| 127 | + }, | |
| 128 | + ); | |
| 129 | + if (input !== undefined) { | |
| 130 | + child.stdin?.write(input); | |
| 131 | + child.stdin?.end(); | |
| 132 | + } | |
| 133 | + }); | |
| 134 | + | |
| 135 | + // Start from an empty index each publish (full file set, not a diff). | |
| 136 | + yield* withIndex(["read-tree", "--empty"]); | |
| 137 | + for (const [path, contents] of files) { | |
| 138 | + const blobHash = (yield* withIndex( | |
| 139 | + ["hash-object", "-w", "--stdin"], | |
| 140 | + Buffer.from(contents, "utf8"), | |
| 141 | + )).trim(); | |
| 142 | + yield* withIndex(["update-index", "--add", "--cacheinfo", `100644,${blobHash},${path}`]); | |
| 143 | + } | |
| 144 | + const treeHash = (yield* withIndex(["write-tree"])).trim(); | |
| 145 | + const commitArgs = ["commit-tree", treeHash, "-m", message]; | |
| 146 | + if (parent) commitArgs.push("-p", parent); | |
| 147 | + const commitEnvVars = { | |
| 148 | + ...process.env, | |
| 149 | + GIT_AUTHOR_NAME: "executor-apps", | |
| 150 | + GIT_AUTHOR_EMAIL: "apps@executor.local", | |
| 151 | + GIT_COMMITTER_NAME: "executor-apps", | |
| 152 | + GIT_COMMITTER_EMAIL: "apps@executor.local", | |
| 153 | + }; | |
| 154 | + const commitHash = (yield* Effect.async<string, ArtifactStoreError>((resume) => { | |
| 155 | + const child = execFile( | |
| 156 | + "git", | |
| 157 | + commitArgs, | |
| 158 | + { cwd: repoDir, encoding: "buffer", env: commitEnvVars }, | |
| 159 | + (error, stdout, stderr) => { | |
| 160 | + if (error) { | |
| 161 | + resume( | |
| 162 | + Effect.fail( | |
| 163 | + new ArtifactStoreError({ | |
| 164 | + message: `git commit-tree failed: ${(stderr as Buffer)?.toString() || error.message}`, | |
| 165 | + cause: error, | |
| 166 | + }), | |
| 167 | + ), | |
| 168 | + ); | |
| 169 | + return; | |
| 170 | + } | |
| 171 | + resume(Effect.succeed((stdout as Buffer).toString("utf8"))); | |
| 172 | + }, | |
| 173 | + ); | |
| 174 | + })).trim(); | |
| 175 | + yield* run(repoDir, ["update-ref", BRANCH, commitHash]); | |
| 176 | + return yield* readMeta(commitHash); | |
| 177 | + }), | |
| 178 | + | |
| 179 | + read: (id: SnapshotId) => | |
| 180 | + run(repoDir, ["ls-tree", "-r", "--name-only", id]).pipe( | |
| 181 | + Effect.flatMap((out) => { | |
| 182 | + const paths = out | |
| 183 | + .split("\n") | |
| 184 | + .map((p) => p.trim()) | |
| 185 | + .filter(Boolean); | |
| 186 | + return Effect.forEach( | |
| 187 | + paths, | |
| 188 | + (path) => | |
| 189 | + run(repoDir, ["cat-file", "blob", `${id}:${path}`]).pipe( | |
| 190 | + Effect.map((contents) => [path, contents] as const), | |
| 191 | + ), | |
| 192 | + { concurrency: 8 }, | |
| 193 | + ); | |
| 194 | + }), | |
| 195 | + Effect.map((entries) => new Map(entries) as FileSet), | |
| 196 | + ), | |
| 197 | + | |
| 198 | + readFile: (id: SnapshotId, path: string) => | |
| 199 | + run(repoDir, ["cat-file", "blob", `${id}:${path}`]).pipe( | |
| 200 | + Effect.map((contents) => contents as string | null), | |
| 201 | + Effect.catchAll(() => Effect.succeed(null)), | |
| 202 | + ), | |
| 203 | + | |
| 204 | + list: (id: SnapshotId) => | |
| 205 | + run(repoDir, ["ls-tree", "-r", "--name-only", id]).pipe( | |
| 206 | + Effect.map((out) => | |
| 207 | + out | |
| 208 | + .split("\n") | |
| 209 | + .map((p) => p.trim()) | |
| 210 | + .filter(Boolean), | |
| 211 | + ), | |
| 212 | + ), | |
| 213 | + | |
| 214 | + latest: () => | |
| 215 | + headCommit().pipe(Effect.flatMap((head) => (head ? readMeta(head) : Effect.succeed(null)))), | |
| 216 | + | |
| 217 | + log: (limit = 50) => | |
| 218 | + headCommit().pipe( | |
| 219 | + Effect.flatMap((head) => { | |
| 220 | + if (!head) return Effect.succeed([] as readonly SnapshotMeta[]); | |
| 221 | + return run(repoDir, [ | |
| 222 | + "log", | |
| 223 | + `--max-count=${limit}`, | |
| 224 | + "--format=%H%x1f%ct%x1f%s", | |
| 225 | + BRANCH, | |
| 226 | + ]).pipe( | |
| 227 | + Effect.map((out) => | |
| 228 | + out | |
| 229 | + .split("\n") | |
| 230 | + .map((line) => line.trim()) | |
| 231 | + .filter(Boolean) | |
| 232 | + .map((line) => { | |
| 233 | + const [hash, committed, subject] = line.split("\x1f"); | |
| 234 | + return { | |
| 235 | + id: asSnapshotId(hash), | |
| 236 | + committedAt: Number(committed) * 1000, | |
| 237 | + message: subject ?? "", | |
| 238 | + } satisfies SnapshotMeta; | |
| 239 | + }), | |
| 240 | + ), | |
| 241 | + ); | |
| 242 | + }), | |
| 243 | + ), | |
| 244 | + }; | |
| 245 | +}; | |
| 246 | + | |
| 247 | +export interface GitArtifactStoreOptions { | |
| 248 | + /** Directory holding one bare repo per scope. Created on demand. */ | |
| 249 | + readonly root: string; | |
| 250 | +} | |
| 251 | + | |
| 252 | +/** Build the git-backed ArtifactStore. Each scope gets a lazily-initialized | |
| 253 | + * bare repo `<root>/<scope>.git`. */ | |
| 254 | +export const makeGitArtifactStore = (options: GitArtifactStoreOptions): ArtifactStore => { | |
| 255 | + const initialized = new Map<string, Promise<ScopeArtifactStore>>(); | |
| 256 | + | |
| 257 | + const init = async (scope: string): Promise<ScopeArtifactStore> => { | |
| 258 | + const safe = sanitizeScope(scope); | |
| 259 | + const repoDir = join(options.root, `${safe}.git`); | |
| 260 | + await mkdir(repoDir, { recursive: true }); | |
| 261 | + await new Promise<void>((resolve, reject) => { | |
| 262 | + execFile("git", ["init", "--bare", "--quiet"], { cwd: repoDir }, (error) => | |
| 263 | + error ? reject(error) : resolve(), | |
| 264 | + ); | |
| 265 | + }); | |
| 266 | + return makeScopeStore(repoDir); | |
| 267 | + }; | |
| 268 | + | |
| 269 | + return { | |
| 270 | + forScope: (scope) => | |
| 271 | + Effect.tryPromise({ | |
| 272 | + try: () => { | |
| 273 | + let existing = initialized.get(scope); | |
| 274 | + if (!existing) { | |
| 275 | + existing = init(scope); | |
| 276 | + initialized.set(scope, existing); | |
| 277 | + } | |
| 278 | + return existing; | |
| 279 | + }, | |
| 280 | + catch: (cause) => | |
| 281 | + new ArtifactStoreError({ message: `failed to open scope repo: ${scope}`, cause }), | |
| 282 | + }), | |
| 283 | + }; | |
| 284 | +}; | |
| @@ -0,0 +1,34 @@ | ||
| 1 | +import { Effect } from "effect"; | |
| 2 | + | |
| 3 | +import type { Invalidation, LiveChannel } from "../seams/live-channel"; | |
| 4 | + | |
| 5 | +// --------------------------------------------------------------------------- | |
| 6 | +// In-process LiveChannel (self-hosted). A per-scope set of listeners; publish | |
| 7 | +// fans out synchronously. The self-host server exposes each subscription as an | |
| 8 | +// SSE stream. The cloud backing (future) makes the storage owner the notifier. | |
| 9 | +// --------------------------------------------------------------------------- | |
| 10 | + | |
| 11 | +export const makeInProcessLiveChannel = (): LiveChannel => { | |
| 12 | + const byScope = new Map<string, Set<(event: Invalidation) => void>>(); | |
| 13 | + | |
| 14 | + return { | |
| 15 | + publish: (event: Invalidation) => | |
| 16 | + Effect.sync(() => { | |
| 17 | + const listeners = byScope.get(event.scope); | |
| 18 | + if (!listeners) return; | |
| 19 | + for (const listener of listeners) listener(event); | |
| 20 | + }), | |
| 21 | + subscribe: (scope: string, listener: (event: Invalidation) => void) => { | |
| 22 | + let set = byScope.get(scope); | |
| 23 | + if (!set) { | |
| 24 | + set = new Set(); | |
| 25 | + byScope.set(scope, set); | |
| 26 | + } | |
| 27 | + set.add(listener); | |
| 28 | + return () => { | |
| 29 | + set?.delete(listener); | |
| 30 | + if (set && set.size === 0) byScope.delete(scope); | |
| 31 | + }; | |
| 32 | + }, | |
| 33 | + }; | |
| 34 | +}; | |
| @@ -0,0 +1,184 @@ | ||
| 1 | +import { mkdirSync } from "node:fs"; | |
| 2 | +import { join, resolve } from "node:path"; | |
| 3 | + | |
| 4 | +import { createClient, type Client } from "@libsql/client"; | |
| 5 | +import { Effect } from "effect"; | |
| 6 | + | |
| 7 | +import { | |
| 8 | + ScopeDbError, | |
| 9 | + type ScopeDb, | |
| 10 | + type ScopeDbHandle, | |
| 11 | + type ScopeWriteEvent, | |
| 12 | +} from "../seams/scope-db"; | |
| 13 | + | |
| 14 | +// --------------------------------------------------------------------------- | |
| 15 | +// libSQL-backed ScopeDb (self-hosted). One SQLite file per scope under | |
| 16 | +// `<root>/<scope>.db`. A control table `__versions` holds a monotonic counter | |
| 17 | +// per user table; every write statement bumps the counters for the tables it | |
| 18 | +// touched and emits a `ScopeWriteEvent` (the runtime forwards that to | |
| 19 | +// LiveChannel). Scope isolation is a separate file per scope — there is no | |
| 20 | +// cross-scope query path. | |
| 21 | +// --------------------------------------------------------------------------- | |
| 22 | + | |
| 23 | +const VERSION_TABLE = "__scope_versions"; | |
| 24 | + | |
| 25 | +// Statements that mutate data. Determines whether a `sql` call bumps versions. | |
| 26 | +const WRITE_RE = /^\s*(insert|update|delete|replace|create|drop|alter)\b/i; | |
| 27 | + | |
| 28 | +// Extract the table names a write statement targets (best-effort: covers the | |
| 29 | +// shapes authored tools produce — INSERT INTO x, UPDATE x, DELETE FROM x, | |
| 30 | +// CREATE TABLE x, REPLACE INTO x). | |
| 31 | +const targetsOf = (sql: string): string[] => { | |
| 32 | + const out = new Set<string>(); | |
| 33 | + const patterns = [ | |
| 34 | + /\binsert\s+(?:or\s+\w+\s+)?into\s+["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 35 | + /\breplace\s+into\s+["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 36 | + /\bupdate\s+["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 37 | + /\bdelete\s+from\s+["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 38 | + /\bcreate\s+table\s+(?:if\s+not\s+exists\s+)?["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 39 | + /\bdrop\s+table\s+(?:if\s+exists\s+)?["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 40 | + /\balter\s+table\s+["'`]?([a-zA-Z_][a-zA-Z0-9_]*)["'`]?/gi, | |
| 41 | + ]; | |
| 42 | + for (const re of patterns) { | |
| 43 | + let m: RegExpExecArray | null; | |
| 44 | + while ((m = re.exec(sql)) !== null) { | |
| 45 | + const name = m[1]; | |
| 46 | + if (name && name !== VERSION_TABLE) out.add(name); | |
| 47 | + } | |
| 48 | + } | |
| 49 | + return [...out]; | |
| 50 | +}; | |
| 51 | + | |
| 52 | +const toArgs = (values: readonly unknown[]): unknown[] => | |
| 53 | + values.map((v) => (v === undefined ? null : v)); | |
| 54 | + | |
| 55 | +const makeHandle = ( | |
| 56 | + scope: string, | |
| 57 | + client: Client, | |
| 58 | + emit: (event: ScopeWriteEvent) => void, | |
| 59 | +): ScopeDbHandle => { | |
| 60 | + const ensureVersionTable = async () => { | |
| 61 | + await client.execute( | |
| 62 | + `CREATE TABLE IF NOT EXISTS ${VERSION_TABLE} (name TEXT PRIMARY KEY, version INTEGER NOT NULL DEFAULT 0)`, | |
| 63 | + ); | |
| 64 | + }; | |
| 65 | + | |
| 66 | + const bump = async (tables: readonly string[]): Promise<{ table: string; version: number }[]> => { | |
| 67 | + const bumped: { table: string; version: number }[] = []; | |
| 68 | + for (const table of tables) { | |
| 69 | + await client.execute({ | |
| 70 | + sql: `INSERT INTO ${VERSION_TABLE} (name, version) VALUES (?, 1) | |
| 71 | + ON CONFLICT(name) DO UPDATE SET version = version + 1`, | |
| 72 | + args: [table], | |
| 73 | + }); | |
| 74 | + const row = await client.execute({ | |
| 75 | + sql: `SELECT version FROM ${VERSION_TABLE} WHERE name = ?`, | |
| 76 | + args: [table], | |
| 77 | + }); | |
| 78 | + bumped.push({ table, version: Number(row.rows[0]?.version ?? 0) }); | |
| 79 | + } | |
| 80 | + return bumped; | |
| 81 | + }; | |
| 82 | + | |
| 83 | + const runStatement = <Row>( | |
| 84 | + sql: string, | |
| 85 | + args: unknown[], | |
| 86 | + ): Effect.Effect<readonly Row[], ScopeDbError> => | |
| 87 | + Effect.tryPromise({ | |
| 88 | + try: async () => { | |
| 89 | + await ensureVersionTable(); | |
| 90 | + const result = await client.execute({ sql, args: args as never }); | |
| 91 | + if (WRITE_RE.test(sql)) { | |
| 92 | + const targets = targetsOf(sql); | |
| 93 | + if (targets.length > 0) { | |
| 94 | + const bumped = await bump(targets); | |
| 95 | + if (bumped.length > 0) emit({ scope, tables: bumped }); | |
| 96 | + } | |
| 97 | + } | |
| 98 | + return result.rows as unknown as readonly Row[]; | |
| 99 | + }, | |
| 100 | + catch: (cause) => | |
| 101 | + new ScopeDbError({ message: `scope-db statement failed for scope ${scope}`, cause }), | |
| 102 | + }); | |
| 103 | + | |
| 104 | + return { | |
| 105 | + sql: <Row = Record<string, unknown>>(strings: TemplateStringsArray, ...values: unknown[]) => { | |
| 106 | + const sql = strings.reduce((acc, part, i) => acc + part + (i < values.length ? "?" : ""), ""); | |
| 107 | + return runStatement<Row>(sql, toArgs(values)); | |
| 108 | + }, | |
| 109 | + exec: <Row = Record<string, unknown>>(sql: string, params: readonly unknown[] = []) => | |
| 110 | + runStatement<Row>(sql, toArgs(params)), | |
| 111 | + tableVersion: (table: string) => | |
| 112 | + Effect.tryPromise({ | |
| 113 | + try: async () => { | |
| 114 | + await ensureVersionTable(); | |
| 115 | + const row = await client.execute({ | |
| 116 | + sql: `SELECT version FROM ${VERSION_TABLE} WHERE name = ?`, | |
| 117 | + args: [table], | |
| 118 | + }); | |
| 119 | + return Number(row.rows[0]?.version ?? 0); | |
| 120 | + }, | |
| 121 | + catch: (cause) => new ScopeDbError({ message: "tableVersion failed", cause }), | |
| 122 | + }), | |
| 123 | + versions: () => | |
| 124 | + Effect.tryPromise({ | |
| 125 | + try: async () => { | |
| 126 | + await ensureVersionTable(); | |
| 127 | + const rows = await client.execute(`SELECT name, version FROM ${VERSION_TABLE}`); | |
| 128 | + const out = new Map<string, number>(); | |
| 129 | + for (const r of rows.rows) out.set(String(r.name), Number(r.version)); | |
| 130 | + return out as ReadonlyMap<string, number>; | |
| 131 | + }, | |
| 132 | + catch: (cause) => new ScopeDbError({ message: "versions failed", cause }), | |
| 133 | + }), | |
| 134 | + }; | |
| 135 | +}; | |
| 136 | + | |
| 137 | +export interface LibsqlScopeDbOptions { | |
| 138 | + /** Directory holding one SQLite file per scope, or ":memory:" for tests. */ | |
| 139 | + readonly root: string; | |
| 140 | +} | |
| 141 | + | |
| 142 | +const toUrl = (path: string): string => (path === ":memory:" ? path : `file:${resolve(path)}`); | |
| 143 | + | |
| 144 | +export const makeLibsqlScopeDb = (options: LibsqlScopeDbOptions): ScopeDb => { | |
| 145 | + const clients = new Map<string, Client>(); | |
| 146 | + const listeners = new Set<(event: ScopeWriteEvent) => void>(); | |
| 147 | + | |
| 148 | + const emit = (event: ScopeWriteEvent) => { | |
| 149 | + for (const listener of listeners) listener(event); | |
| 150 | + }; | |
| 151 | + | |
| 152 | + const clientFor = (scope: string): Client => { | |
| 153 | + let client = clients.get(scope); | |
| 154 | + if (!client) { | |
| 155 | + if (options.root === ":memory:") { | |
| 156 | + client = createClient({ url: ":memory:" }); | |
| 157 | + } else { | |
| 158 | + mkdirSync(options.root, { recursive: true }); | |
| 159 | + const safe = scope.replace(/[^a-zA-Z0-9._-]/g, "_"); | |
| 160 | + client = createClient({ url: toUrl(join(options.root, `${safe}.db`)) }); | |
| 161 | + } | |
| 162 | + clients.set(scope, client); | |
| 163 | + } | |
| 164 | + return client; | |
| 165 | + }; | |
| 166 | + | |
| 167 | + return { | |
| 168 | + forScope: (scope) => | |
| 169 | + Effect.try({ | |
| 170 | + try: () => makeHandle(scope, clientFor(scope), emit), | |
| 171 | + catch: (cause) => new ScopeDbError({ message: `failed to open scope db ${scope}`, cause }), | |
| 172 | + }), | |
| 173 | + onWrite: (listener) => { | |
| 174 | + listeners.add(listener); | |
| 175 | + return () => listeners.delete(listener); | |
| 176 | + }, | |
| 177 | + close: () => | |
| 178 | + Effect.sync(() => { | |
| 179 | + for (const client of clients.values()) client.close(); | |
| 180 | + clients.clear(); | |
| 181 | + listeners.clear(); | |
| 182 | + }), | |
| 183 | + }; | |
| 184 | +}; | |
| @@ -0,0 +1,125 @@ | ||
| 1 | +import { describe, expect, it } from "vitest"; | |
| 2 | +import { Effect } from "effect"; | |
| 3 | + | |
| 4 | +import { bundleEntry } from "../pipeline/bundle"; | |
| 5 | +import { makeQuickjsToolSandbox } from "./quickjs-tool-sandbox"; | |
| 6 | +import { ISSUES_SYNC_TS, SEARCH_ALL_MAIL_TS } from "../testing/daily-brief"; | |
| 7 | +import type { HandleBridge } from "../seams/tool-sandbox"; | |
| 8 | + | |
| 9 | +const run = <A, E>(effect: Effect.Effect<A, E>): Promise<A> => Effect.runPromise(effect); | |
| 10 | + | |
| 11 | +describe("QuickJS ToolSandbox", () => { | |
| 12 | + const sandbox = makeQuickjsToolSandbox(); | |
| 13 | + | |
| 14 | + it("bundles + collects a tool descriptor (deterministic)", async () => { | |
| 15 | + const files = new Map([["tools/issues-sync.ts", ISSUES_SYNC_TS]]); | |
| 16 | + const { code } = await run(bundleEntry({ files, entry: "tools/issues-sync.ts" })); | |
| 17 | + const result = await run(sandbox.collect(code)); | |
| 18 | + const descriptor = result.artifacts.default.descriptor as { | |
| 19 | + kind: string; | |
| 20 | + connections: Record<string, { decl: string; integration: string }>; | |
| 21 | + inputJsonSchema: { type: string; properties: Record<string, unknown> }; | |
| 22 | + hasHandler: boolean; | |
| 23 | + }; | |
| 24 | + expect(descriptor.kind).toBe("tool"); | |
| 25 | + expect(descriptor.connections.github).toEqual( | |
| 26 | + expect.objectContaining({ decl: "single", integration: "github" }), | |
| 27 | + ); | |
| 28 | + expect(descriptor.hasHandler).toBe(true); | |
| 29 | + expect(descriptor.inputJsonSchema.type).toBe("object"); | |
| 30 | + expect(descriptor.inputJsonSchema.properties).toHaveProperty("repos"); | |
| 31 | + expect(descriptor.inputJsonSchema.properties).toHaveProperty("since"); | |
| 32 | + }); | |
| 33 | + | |
| 34 | + it("collects a fan-out (connections) declaration", async () => { | |
| 35 | + const files = new Map([["tools/search-all-mail.ts", SEARCH_ALL_MAIL_TS]]); | |
| 36 | + const { code } = await run(bundleEntry({ files, entry: "tools/search-all-mail.ts" })); | |
| 37 | + const result = await run(sandbox.collect(code)); | |
| 38 | + const descriptor = result.artifacts.default.descriptor as { | |
| 39 | + connections: Record<string, { decl: string; integration: string }>; | |
| 40 | + }; | |
| 41 | + expect(descriptor.connections.inboxes).toEqual( | |
| 42 | + expect.objectContaining({ decl: "array", integration: "gmail" }), | |
| 43 | + ); | |
| 44 | + }); | |
| 45 | + | |
| 46 | + it("invokes a tool, routing injected-client + db calls through the bridge", async () => { | |
| 47 | + const files = new Map([["tools/issues-sync.ts", ISSUES_SYNC_TS]]); | |
| 48 | + const { code } = await run(bundleEntry({ files, entry: "tools/issues-sync.ts" })); | |
| 49 | + | |
| 50 | + const calls: { root: string; path: readonly string[]; args: readonly unknown[] }[] = []; | |
| 51 | + const bridge: HandleBridge = { | |
| 52 | + call: ({ root, path, args }) => | |
| 53 | + Effect.sync(() => { | |
| 54 | + calls.push({ root, path, args }); | |
| 55 | + const key = `${root}.${path.join(".")}`; | |
| 56 | + if (key === "github.repos.listForAuthenticatedUser") { | |
| 57 | + return [{ full_name: "acme/app" }]; | |
| 58 | + } | |
| 59 | + if (key === "github.issues.listForRepo") { | |
| 60 | + return [ | |
| 61 | + { | |
| 62 | + number: 1, | |
| 63 | + title: "Bug", | |
| 64 | + labels: [{ name: "bug" }], | |
| 65 | + assignee: { login: "rhys" }, | |
| 66 | + updated_at: "2026-01-01T00:00:00Z", | |
| 67 | + html_url: "https://github.com/acme/app/issues/1", | |
| 68 | + }, | |
| 69 | + ]; | |
| 70 | + } | |
| 71 | + // db.sql calls return empty rows | |
| 72 | + return []; | |
| 73 | + }), | |
| 74 | + }; | |
| 75 | + | |
| 76 | + const result = await run( | |
| 77 | + sandbox.invoke( | |
| 78 | + code, | |
| 79 | + { | |
| 80 | + artifact: "issues-sync", | |
| 81 | + kind: "tool", | |
| 82 | + input: {}, | |
| 83 | + roots: { github: { kind: "single" }, db: { kind: "single" } }, | |
| 84 | + }, | |
| 85 | + bridge, | |
| 86 | + ), | |
| 87 | + ); | |
| 88 | + | |
| 89 | + expect(result.output).toEqual({ synced: 1, repos: 1 }); | |
| 90 | + // The db write went through the bridge (root "db"). | |
| 91 | + expect(calls.some((c) => c.root === "db")).toBe(true); | |
| 92 | + expect(calls.some((c) => c.root === "github")).toBe(true); | |
| 93 | + }); | |
| 94 | + | |
| 95 | + it("rejects a non-deterministic descriptor (Math.random at describe time)", async () => { | |
| 96 | + const nondeterministic = `import { defineTool } from "executor:app"; | |
| 97 | +import { z } from "zod"; | |
| 98 | +export default defineTool({ | |
| 99 | + description: "x-" + Math.random(), | |
| 100 | + input: z.object({}), | |
| 101 | + async handler() { return {}; }, | |
| 102 | +});`; | |
| 103 | + const files = new Map([["tools/rng.ts", nondeterministic]]); | |
| 104 | + const { code } = await run(bundleEntry({ files, entry: "tools/rng.ts" })); | |
| 105 | + const exit = await Effect.runPromiseExit(sandbox.collect(code)); | |
| 106 | + expect(exit._tag).toBe("Failure"); | |
| 107 | + }); | |
| 108 | + | |
| 109 | + it("denies network (fetch throws in the sandbox)", async () => { | |
| 110 | + const usesFetch = `import { defineTool } from "executor:app"; | |
| 111 | +import { z } from "zod"; | |
| 112 | +export default defineTool({ | |
| 113 | + description: "fetches", | |
| 114 | + input: z.object({}), | |
| 115 | + async handler() { await fetch("https://example.com"); return {}; }, | |
| 116 | +});`; | |
| 117 | + const files = new Map([["tools/net.ts", usesFetch]]); | |
| 118 | + const { code } = await run(bundleEntry({ files, entry: "tools/net.ts" })); | |
| 119 | + const bridge: HandleBridge = { call: () => Effect.succeed(null) }; | |
| 120 | + const exit = await Effect.runPromiseExit( | |
| 121 | + sandbox.invoke(code, { artifact: "net", kind: "tool", input: {}, roots: {} }, bridge), | |
| 122 | + ); | |
| 123 | + expect(exit._tag).toBe("Failure"); | |
| 124 | + }); | |
| 125 | +}); | |
| @@ -0,0 +1,252 @@ | ||
| 1 | +import { Effect } from "effect"; | |
| 2 | +import type { SandboxToolInvoker } from "@executor-js/codemode-core"; | |
| 3 | +import { makeQuickJsExecutor } from "@executor-js/runtime-quickjs"; | |
| 4 | + | |
| 5 | +import { | |
| 6 | + ToolSandboxError, | |
| 7 | + type CollectResult, | |
| 8 | + type HandleBridge, | |
| 9 | + type InvokeRequest, | |
| 10 | + type InvokeResult, | |
| 11 | + type ToolSandbox, | |
| 12 | +} from "../seams/tool-sandbox"; | |
| 13 | + | |
| 14 | +// --------------------------------------------------------------------------- | |
| 15 | +// QuickJS-backed ToolSandbox (self-hosted). | |
| 16 | +// | |
| 17 | +// The published bundle is a CJS string. The sandbox body prepends a `require` | |
| 18 | +// shim providing `executor:app` (defineTool/defineWorkflow/connection/ | |
| 19 | +// connections/catalog) and `executor:ui` stubs, then executes the bundle so | |
| 20 | +// `module.exports.default` is the artifact. A driver appended after the bundle | |
| 21 | +// either collects descriptors (nothing bound) or runs one handler with injected | |
| 22 | +// clients. | |
| 23 | +// | |
| 24 | +// Injected clients are Proxies that turn `github.issues.listForRepo(args)` into | |
| 25 | +// `await __invokeTool("__handle__", { root, path, args })` — the ONE bridge the | |
| 26 | +// QuickJS runtime already provides (`tools`/`__invokeTool`). Our | |
| 27 | +// `SandboxToolInvoker` decodes that and forwards to the host `HandleBridge`. | |
| 28 | +// Everything crossing is JSON (the cloud version is RPC), so the interface | |
| 29 | +// stays honest. | |
| 30 | +// | |
| 31 | +// Determinism: `collect` runs the collection body twice and byte-compares the | |
| 32 | +// descriptor JSON. Effectful top-levels (Math.random, Date.now) diverge and are | |
| 33 | +// rejected. QuickJS denies `fetch` and enforces a deadline + memory cap. | |
| 34 | +// --------------------------------------------------------------------------- | |
| 35 | + | |
| 36 | +const COLLECT_TIMEOUT_MS = 10_000; | |
| 37 | +const INVOKE_TIMEOUT_MS = 30_000; | |
| 38 | + | |
| 39 | +// The shim + module system injected before the bundle. Kept as a plain string | |
| 40 | +// (QuickJS evals a string). `defineTool`/`defineWorkflow` record their def so | |
| 41 | +// the driver can read it back. Clients are built by `__mkHandle`. | |
| 42 | +const runtimePrelude = ` | |
| 43 | +var __modules = {}; | |
| 44 | +var __defs = { tool: null, workflow: null, ui: null }; | |
| 45 | +function __recordDefault(mod) { return mod; } | |
| 46 | +var __handleBridge = function(root, path, args) { | |
| 47 | + // Route every injected-client method call through the single tool bridge. | |
| 48 | + return tools.__handle__({ root: root, path: path, args: args }); | |
| 49 | +}; | |
| 50 | +function __mkHandle(root, prefix) { | |
| 51 | + var target = function(){}; | |
| 52 | + return new Proxy(target, { | |
| 53 | + get: function(_t, prop) { | |
| 54 | + if (prop === 'then' || typeof prop === 'symbol') return undefined; | |
| 55 | + return __mkHandle(root, prefix.concat([String(prop)])); | |
| 56 | + }, | |
| 57 | + apply: function(_t, _this, callArgs) { | |
| 58 | + return __handleBridge(root, prefix, callArgs); | |
| 59 | + } | |
| 60 | + }); | |
| 61 | +} | |
| 62 | +var __executorApp = { | |
| 63 | + connection: function(integration, opts) { return { __decl: 'single', integration: integration, description: opts && opts.description }; }, | |
| 64 | + connections: function(integration, opts) { return { __decl: 'array', integration: integration, description: opts && opts.description }; }, | |
| 65 | + catalog: function() { return { __decl: 'catalog' }; }, | |
| 66 | + defineTool: function(def) { __defs.tool = def; return def; }, | |
| 67 | + defineWorkflow: function(def) { __defs.workflow = def; return def; }, | |
| 68 | +}; | |
| 69 | +var __executorUi = { | |
| 70 | + config: function(){ return undefined; }, | |
| 71 | + useQuery: function(){ return { data: [], isLoading: false, refetch: function(){} }; }, | |
| 72 | + useTool: function(){ return { run: function(){ return Promise.resolve(); }, isRunning: false }; }, | |
| 73 | +}; | |
| 74 | +function __require(id) { | |
| 75 | + if (id === 'executor:app') return __executorApp; | |
| 76 | + if (id === 'executor:ui') return __executorUi; | |
| 77 | + if (id === 'executor:ui/components') return {}; | |
| 78 | + throw new Error('module not available in sandbox: ' + id); | |
| 79 | +} | |
| 80 | +`; | |
| 81 | + | |
| 82 | +// Run the CJS bundle. The bundle's virtual entry sets `globalThis.__artifact` | |
| 83 | +// (the def object returned by define*), `globalThis.__zodToJson` and | |
| 84 | +// `__zodToJsonOutput` (bound to the author's inlined zod). `require` is our | |
| 85 | +// shim; `defineTool`/`defineWorkflow` also record into `__defs` as a fallback. | |
| 86 | +const wrapBundle = (bundle: string): string => ` | |
| 87 | +var module = { exports: {} }; | |
| 88 | +var exports = module.exports; | |
| 89 | +var require = __require; | |
| 90 | +(function(module, exports, require){ | |
| 91 | +${bundle} | |
| 92 | +})(module, exports, require); | |
| 93 | +`; | |
| 94 | + | |
| 95 | +// Collect driver: describe the artifact's connections + input/output schema. | |
| 96 | +// Deterministic JSON only — no handler execution. | |
| 97 | +const collectDriver = ` | |
| 98 | +return await (async () => { | |
| 99 | + var def = __defs.tool || __defs.workflow || (globalThis.__artifact && (globalThis.__artifact.default || globalThis.__artifact)); | |
| 100 | + var kind = __defs.workflow ? 'workflow' : 'tool'; | |
| 101 | + var conns = {}; | |
| 102 | + if (def && def.connections) { | |
| 103 | + for (var k in def.connections) { | |
| 104 | + var c = def.connections[k]; | |
| 105 | + conns[k] = { decl: c && c.__decl ? c.__decl : 'single', integration: c && c.integration, description: c && c.description }; | |
| 106 | + } | |
| 107 | + } | |
| 108 | + var toJson = (typeof globalThis.__zodToJson === 'function') ? globalThis.__zodToJson : function(){ return undefined; }; | |
| 109 | + var toJsonOut = (typeof globalThis.__zodToJsonOutput === 'function') ? globalThis.__zodToJsonOutput : function(){ return undefined; }; | |
| 110 | + return { | |
| 111 | + kind: kind, | |
| 112 | + description: def && def.description, | |
| 113 | + connections: conns, | |
| 114 | + annotations: def && def.annotations, | |
| 115 | + schedule: def && def.schedule, | |
| 116 | + hasHandler: !!(def && (def.handler || def.run)), | |
| 117 | + inputJsonSchema: def ? toJson(def.input) : undefined, | |
| 118 | + outputJsonSchema: def ? toJsonOut(def.output) : undefined, | |
| 119 | + }; | |
| 120 | +})() | |
| 121 | +`; | |
| 122 | + | |
| 123 | +const buildCollectCode = (bundle: string): string => | |
| 124 | + runtimePrelude + wrapBundle(bundle) + collectDriver; | |
| 125 | + | |
| 126 | +// Invoke driver: build injected clients from the request roots, then call the | |
| 127 | +// artifact's handler with (input, injected). Fan-out arrays become arrays of | |
| 128 | +// element handles. | |
| 129 | +const buildInvokeDriver = (request: InvokeRequest): string => { | |
| 130 | + const rootsLiteral = JSON.stringify(request.roots); | |
| 131 | + const inputLiteral = JSON.stringify(request.input ?? {}); | |
| 132 | + return ` | |
| 133 | +return await (async () => { | |
| 134 | + var def = __defs.tool || (globalThis.__artifact && (globalThis.__artifact.default || globalThis.__artifact)); | |
| 135 | + if (!def || typeof def.handler !== 'function') throw new Error('artifact has no handler: ${request.artifact}'); | |
| 136 | + var roots = ${rootsLiteral}; | |
| 137 | + var injected = {}; | |
| 138 | + for (var name in roots) { | |
| 139 | + var spec = roots[name]; | |
| 140 | + if (spec.kind === 'array') { | |
| 141 | + var arr = []; | |
| 142 | + for (var i = 0; i < spec.count; i++) arr.push(__mkHandle(name + '#' + i, [])); | |
| 143 | + injected[name] = arr; | |
| 144 | + } else { | |
| 145 | + injected[name] = __mkHandle(name, []); | |
| 146 | + } | |
| 147 | + } | |
| 148 | + var out = await def.handler(${inputLiteral}, injected); | |
| 149 | + return out; | |
| 150 | +})() | |
| 151 | +`; | |
| 152 | +}; | |
| 153 | + | |
| 154 | +const buildInvokeCode = (bundle: string, request: InvokeRequest): string => | |
| 155 | + runtimePrelude + wrapBundle(bundle) + buildInvokeDriver(request); | |
| 156 | + | |
| 157 | +// A no-op invoker for collect: no handle calls should happen; if they do | |
| 158 | +// (misbehaving describe path), fail loudly. | |
| 159 | +const collectInvoker: SandboxToolInvoker = { | |
| 160 | + invoke: () => Effect.fail(new Error("collect must not make handle calls")) as never, | |
| 161 | +}; | |
| 162 | + | |
| 163 | +export interface QuickjsToolSandboxOptions { | |
| 164 | + readonly collectTimeoutMs?: number; | |
| 165 | + readonly invokeTimeoutMs?: number; | |
| 166 | +} | |
| 167 | + | |
| 168 | +export const makeQuickjsToolSandbox = (options: QuickjsToolSandboxOptions = {}): ToolSandbox => { | |
| 169 | + const collectExecutor = makeQuickJsExecutor({ | |
| 170 | + timeoutMs: options.collectTimeoutMs ?? COLLECT_TIMEOUT_MS, | |
| 171 | + }); | |
| 172 | + const invokeExecutor = makeQuickJsExecutor({ | |
| 173 | + timeoutMs: options.invokeTimeoutMs ?? INVOKE_TIMEOUT_MS, | |
| 174 | + }); | |
| 175 | + | |
| 176 | + const runCollect = (bundle: string): Effect.Effect<unknown, ToolSandboxError> => | |
| 177 | + collectExecutor.execute(buildCollectCode(bundle), collectInvoker).pipe( | |
| 178 | + Effect.mapError( | |
| 179 | + (cause) => new ToolSandboxError({ kind: "collect", message: "collect run failed", cause }), | |
| 180 | + ), | |
| 181 | + Effect.flatMap((result) => { | |
| 182 | + if (result.error) { | |
| 183 | + return Effect.fail(new ToolSandboxError({ kind: "collect", message: result.error })); | |
| 184 | + } | |
| 185 | + return Effect.succeed(result.result); | |
| 186 | + }), | |
| 187 | + ); | |
| 188 | + | |
| 189 | + return { | |
| 190 | + collect: (bundle: string) => | |
| 191 | + Effect.gen(function* () { | |
| 192 | + // Run twice, byte-compare (determinism gate). | |
| 193 | + const first = yield* runCollect(bundle); | |
| 194 | + const second = yield* runCollect(bundle); | |
| 195 | + const a = JSON.stringify(first); | |
| 196 | + const b = JSON.stringify(second); | |
| 197 | + if (a !== b) { | |
| 198 | + return yield* Effect.fail( | |
| 199 | + new ToolSandboxError({ | |
| 200 | + kind: "nondeterministic", | |
| 201 | + message: | |
| 202 | + "descriptor collection is non-deterministic (an artifact read Math.random/Date.now or otherwise diverged between runs)", | |
| 203 | + }), | |
| 204 | + ); | |
| 205 | + } | |
| 206 | + const descriptor = first as { kind?: string }; | |
| 207 | + const kind = descriptor?.kind === "workflow" ? "workflow" : "tool"; | |
| 208 | + const result: CollectResult = { | |
| 209 | + artifacts: { | |
| 210 | + [String((descriptor as { artifact?: string }).artifact ?? "default")]: { | |
| 211 | + kind, | |
| 212 | + descriptor: first, | |
| 213 | + }, | |
| 214 | + }, | |
| 215 | + }; | |
| 216 | + return result; | |
| 217 | + }), | |
| 218 | + | |
| 219 | + invoke: (bundle: string, request: InvokeRequest, bridge: HandleBridge) => | |
| 220 | + Effect.gen(function* () { | |
| 221 | + // The invoker decodes the routed handle call and forwards to the host | |
| 222 | + // bridge. Path 0 is `__handle__`; the single arg is {root, path, args}. | |
| 223 | + const invoker: SandboxToolInvoker = { | |
| 224 | + invoke: (input: { path: string; args: unknown }) => { | |
| 225 | + if (input.path !== "__handle__") { | |
| 226 | + return Effect.fail(new Error(`unexpected sandbox call: ${input.path}`)) as never; | |
| 227 | + } | |
| 228 | + const call = input.args as { | |
| 229 | + root: string; | |
| 230 | + path: readonly string[]; | |
| 231 | + args: readonly unknown[]; | |
| 232 | + }; | |
| 233 | + return bridge.call({ root: call.root, path: call.path, args: call.args }) as never; | |
| 234 | + }, | |
| 235 | + }; | |
| 236 | + const result = yield* invokeExecutor | |
| 237 | + .execute(buildInvokeCode(bundle, request), invoker) | |
| 238 | + .pipe( | |
| 239 | + Effect.mapError( | |
| 240 | + (cause) => | |
| 241 | + new ToolSandboxError({ kind: "invoke", message: "invoke run failed", cause }), | |
| 242 | + ), | |
| 243 | + ); | |
| 244 | + if (result.error) { | |
| 245 | + return yield* Effect.fail( | |
| 246 | + new ToolSandboxError({ kind: "invoke", message: result.error }), | |
| 247 | + ); | |
| 248 | + } | |
| 249 | + return { output: result.result, logs: result.logs ?? [] } satisfies InvokeResult; | |
| 250 | + }), | |
| 251 | + }; | |
| 252 | +}; | |
| @@ -0,0 +1,212 @@ | ||
| 1 | +import { build } from "esbuild"; | |
| 2 | + | |
| 3 | +import { Effect } from "effect"; | |
| 4 | + | |
| 5 | +import { ToolSandboxError } from "../seams/tool-sandbox"; | |
| 6 | + | |
| 7 | +// --------------------------------------------------------------------------- | |
| 8 | +// bundle — the FDI pipeline's bundle stage. | |
| 9 | +// | |
| 10 | +// One esbuild pass per artifact entry. Platform modules (`executor:app`, | |
| 11 | +// `executor:ui`, react, recharts, ...) stay EXTERNAL — the sandbox shim | |
| 12 | +// provides the real behavior. A small allowlist of schema libraries (`zod`, | |
| 13 | +// `@standard-schema/spec`) is INLINED from node_modules so the author's schema | |
| 14 | +// runs in the sandbox AND its `toJSONSchema` is reachable to extract the | |
| 15 | +// descriptor's input/output JSON Schema. Every other bare import is rejected | |
| 16 | +// with a diagnostic: npm deps in user bundles are out of scope. | |
| 17 | +// | |
| 18 | +// Output is a single CJS string with the externals left as `require(...)` | |
| 19 | +// calls the sandbox resolves. | |
| 20 | +// --------------------------------------------------------------------------- | |
| 21 | + | |
| 22 | +/** Modules the platform provides inside the sandbox (kept external, resolved by | |
| 23 | + * the sandbox `require` shim). */ | |
| 24 | +export const PLATFORM_MODULES = new Set<string>([ | |
| 25 | + "executor:app", | |
| 26 | + "executor:ui", | |
| 27 | + "executor:ui/components", | |
| 28 | + "react", | |
| 29 | + "react/jsx-runtime", | |
| 30 | + "react-dom", | |
| 31 | + "recharts", | |
| 32 | + "lucide-react", | |
| 33 | +]); | |
| 34 | + | |
| 35 | +/** npm modules we deliberately inline (schema runtime that must run in the | |
| 36 | + * sandbox). Everything else bare is rejected. */ | |
| 37 | +export const INLINABLE_MODULES = new Set<string>(["zod", "@standard-schema/spec"]); | |
| 38 | + | |
| 39 | +const isInlinable = (path: string): boolean => { | |
| 40 | + if (INLINABLE_MODULES.has(path)) return true; | |
| 41 | + // zod subpaths (`zod/v4`, `zod/mini`) are fine too. | |
| 42 | + return [...INLINABLE_MODULES].some((m) => path === m || path.startsWith(`${m}/`)); | |
| 43 | +}; | |
| 44 | + | |
| 45 | +export interface BundleInput { | |
| 46 | + readonly files: ReadonlyMap<string, string>; | |
| 47 | + readonly entry: string; | |
| 48 | +} | |
| 49 | + | |
| 50 | +export interface BundleOutput { | |
| 51 | + readonly code: string; | |
| 52 | +} | |
| 53 | + | |
| 54 | +const resolveRelative = (base: string, rel: string): string => { | |
| 55 | + const parts = (base ? base.split("/") : []).concat(rel.split("/")); | |
| 56 | + const out: string[] = []; | |
| 57 | + for (const part of parts) { | |
| 58 | + if (part === "" || part === ".") continue; | |
| 59 | + if (part === "..") out.pop(); | |
| 60 | + else out.push(part); | |
| 61 | + } | |
| 62 | + return out.join("/"); | |
| 63 | +}; | |
| 64 | + | |
| 65 | +const candidates = (path: string): string[] => { | |
| 66 | + if (/\.(tsx?|jsx?)$/.test(path)) return [path]; | |
| 67 | + return [ | |
| 68 | + path, | |
| 69 | + `${path}.ts`, | |
| 70 | + `${path}.tsx`, | |
| 71 | + `${path}.js`, | |
| 72 | + `${path}.jsx`, | |
| 73 | + `${path}/index.ts`, | |
| 74 | + `${path}/index.tsx`, | |
| 75 | + ]; | |
| 76 | +}; | |
| 77 | + | |
| 78 | +const FILESET_NS = "fileset"; | |
| 79 | +const VIRTUAL_ENTRY = "executor-apps://entry"; | |
| 80 | + | |
| 81 | +/** The virtual entry esbuild starts from: re-export the author entry AND expose | |
| 82 | + * the shared zod instance as a global so the sandbox collect driver can call | |
| 83 | + * `toJSONSchema` against the SAME zod the author bundled. */ | |
| 84 | +const virtualEntrySource = (authorEntry: string) => ` | |
| 85 | +import __artifactModule from ${JSON.stringify(`/${authorEntry}`)}; | |
| 86 | +import * as __zod from "zod"; | |
| 87 | +globalThis.__artifact = __artifactModule; | |
| 88 | +globalThis.__zodToJson = function (schema) { | |
| 89 | + if (!schema) return undefined; | |
| 90 | + try { | |
| 91 | + return __zod.toJSONSchema(schema, { io: "input", unrepresentable: "any" }); | |
| 92 | + } catch (e) { | |
| 93 | + try { return __zod.toJSONSchema(schema); } catch (e2) { return {}; } | |
| 94 | + } | |
| 95 | +}; | |
| 96 | +globalThis.__zodToJsonOutput = function (schema) { | |
| 97 | + if (!schema) return undefined; | |
| 98 | + try { | |
| 99 | + return __zod.toJSONSchema(schema, { io: "output", unrepresentable: "any" }); | |
| 100 | + } catch (e) { | |
| 101 | + try { return __zod.toJSONSchema(schema); } catch (e2) { return {}; } | |
| 102 | + } | |
| 103 | +}; | |
| 104 | +`; | |
| 105 | + | |
| 106 | +const fileSetPlugin = (files: ReadonlyMap<string, string>, authorEntry: string) => ({ | |
| 107 | + name: "executor-apps-fileset", | |
| 108 | + // eslint-disable-next-line @typescript-eslint/no-explicit-any | |
| 109 | + setup(build2: any) { | |
| 110 | + const norm = (p: string) => p.replace(/^\.\//, "").replace(/\/\.\//g, "/"); | |
| 111 | + | |
| 112 | + build2.onResolve( | |
| 113 | + { filter: /.*/ }, | |
| 114 | + (args: { path: string; importer: string; namespace?: string; resolveDir?: string }) => { | |
| 115 | + if (args.path === VIRTUAL_ENTRY) { | |
| 116 | + return { path: VIRTUAL_ENTRY, namespace: "virtual" }; | |
| 117 | + } | |
| 118 | + // Imports originating OUTSIDE the file set (from inlined npm modules | |
| 119 | + // like zod, or the virtual entry's `import "zod"`) are left to | |
| 120 | + // esbuild's default node resolution. We only govern file-set imports. | |
| 121 | + const fromFileSet = args.namespace === FILESET_NS || args.namespace === "virtual"; | |
| 122 | + if (!fromFileSet) return undefined; | |
| 123 | + | |
| 124 | + if (PLATFORM_MODULES.has(args.path)) { | |
| 125 | + return { path: args.path, external: true }; | |
| 126 | + } | |
| 127 | + if (isInlinable(args.path)) { | |
| 128 | + // Inline from node_modules via default resolution. | |
| 129 | + return undefined; | |
| 130 | + } | |
| 131 | + // Absolute (virtual entry uses `/tools/x.ts`) or relative -> file set. | |
| 132 | + if (args.path.startsWith("/")) { | |
| 133 | + const joined = norm(args.path.slice(1)); | |
| 134 | + for (const candidate of candidates(joined)) { | |
| 135 | + if (files.has(candidate)) return { path: candidate, namespace: FILESET_NS }; | |
| 136 | + } | |
| 137 | + return { errors: [{ text: `cannot resolve "${args.path}" in the app file set` }] }; | |
| 138 | + } | |
| 139 | + if (args.path.startsWith(".")) { | |
| 140 | + const base = args.importer.split("/").slice(0, -1).join("/"); | |
| 141 | + const joined = norm(resolveRelative(base, args.path)); | |
| 142 | + for (const candidate of candidates(joined)) { | |
| 143 | + if (files.has(candidate)) return { path: candidate, namespace: FILESET_NS }; | |
| 144 | + } | |
| 145 | + return { | |
| 146 | + errors: [ | |
| 147 | + { text: `cannot resolve "${args.path}" from "${args.importer}" in the app file set` }, | |
| 148 | + ], | |
| 149 | + }; | |
| 150 | + } | |
| 151 | + return { | |
| 152 | + errors: [ | |
| 153 | + { | |
| 154 | + text: `bare import "${args.path}" is not allowed: apps may only import platform modules (${[...PLATFORM_MODULES].join(", ")}), the schema runtime (zod), or files within the scope`, | |
| 155 | + }, | |
| 156 | + ], | |
| 157 | + }; | |
| 158 | + }, | |
| 159 | + ); | |
| 160 | + | |
| 161 | + build2.onLoad({ filter: /.*/, namespace: "virtual" }, () => ({ | |
| 162 | + contents: virtualEntrySource(authorEntry), | |
| 163 | + loader: "ts", | |
| 164 | + resolveDir: process.cwd(), | |
| 165 | + })); | |
| 166 | + | |
| 167 | + build2.onLoad({ filter: /.*/, namespace: FILESET_NS }, (args: { path: string }) => { | |
| 168 | + const contents = files.get(args.path); | |
| 169 | + if (contents === undefined) { | |
| 170 | + return { errors: [{ text: `missing file in set: ${args.path}` }] }; | |
| 171 | + } | |
| 172 | + const loader = args.path.endsWith(".tsx") | |
| 173 | + ? "tsx" | |
| 174 | + : args.path.endsWith(".ts") | |
| 175 | + ? "ts" | |
| 176 | + : args.path.endsWith(".jsx") | |
| 177 | + ? "jsx" | |
| 178 | + : "js"; | |
| 179 | + return { contents, loader, resolveDir: process.cwd() }; | |
| 180 | + }); | |
| 181 | + }, | |
| 182 | +}); | |
| 183 | + | |
| 184 | +/** Bundle one entry from the file set to a single CJS string. */ | |
| 185 | +export const bundleEntry = (input: BundleInput): Effect.Effect<BundleOutput, ToolSandboxError> => | |
| 186 | + Effect.tryPromise({ | |
| 187 | + try: async () => { | |
| 188 | + const result = await build({ | |
| 189 | + entryPoints: [VIRTUAL_ENTRY], | |
| 190 | + bundle: true, | |
| 191 | + write: false, | |
| 192 | + format: "cjs", | |
| 193 | + platform: "neutral", | |
| 194 | + target: "es2022", | |
| 195 | + minify: false, | |
| 196 | + treeShaking: true, | |
| 197 | + jsx: "automatic", | |
| 198 | + logLevel: "silent", | |
| 199 | + external: [...PLATFORM_MODULES], | |
| 200 | + plugins: [fileSetPlugin(input.files, input.entry) as never], | |
| 201 | + }); | |
| 202 | + const out = result.outputFiles?.[0]?.text; | |
| 203 | + if (out === undefined) throw new Error("esbuild produced no output"); | |
| 204 | + return { code: out }; | |
| 205 | + }, | |
| 206 | + catch: (cause) => | |
| 207 | + new ToolSandboxError({ | |
| 208 | + kind: "bundle", | |
| 209 | + message: `bundle failed for ${input.entry}: ${cause instanceof Error ? cause.message : String(cause)}`, | |
| 210 | + cause, | |
| 211 | + }), | |
| 212 | + }); | |
| @@ -0,0 +1,60 @@ | ||
| 1 | +import type { Effect } from "effect"; | |
| 2 | +import { Data } from "effect"; | |
| 3 | + | |
| 4 | +// --------------------------------------------------------------------------- | |
| 5 | +// ArtifactStore — the per-scope git-backed source store. | |
| 6 | +// | |
| 7 | +// A scope's source lives as a bare git repository; a publish writes a commit | |
| 8 | +// and the commit hash IS the snapshot id (immutable, content-addressed by git). | |
| 9 | +// The self-hosted backing is a bare git repo per scope on disk (git CLI via | |
| 10 | +// subprocess). The cloud backing (future) is Cloudflare Artifacts. The seam is | |
| 11 | +// substrate-neutral: read/write/list/latest/log over a flat file set, plus a | |
| 12 | +// snapshot immutability guarantee (a committed snapshot's bytes never change). | |
| 13 | +// --------------------------------------------------------------------------- | |
| 14 | + | |
| 15 | +/** A published snapshot id. In the git backing this is the commit hash. */ | |
| 16 | +export type SnapshotId = string & { readonly __snapshotId: unique symbol }; | |
| 17 | + | |
| 18 | +export const asSnapshotId = (value: string): SnapshotId => value as SnapshotId; | |
| 19 | + | |
| 20 | +/** A flat set of source files, path -> UTF-8 contents. Paths are POSIX, | |
| 21 | + * relative to the scope root (e.g. `tools/issues-sync.ts`). */ | |
| 22 | +export type FileSet = ReadonlyMap<string, string>; | |
| 23 | + | |
| 24 | +export interface SnapshotMeta { | |
| 25 | + readonly id: SnapshotId; | |
| 26 | + readonly message: string; | |
| 27 | + /** Epoch ms the snapshot was written. */ | |
| 28 | + readonly committedAt: number; | |
| 29 | +} | |
| 30 | + | |
| 31 | +export class ArtifactStoreError extends Data.TaggedError("ArtifactStoreError")<{ | |
| 32 | + readonly message: string; | |
| 33 | + readonly cause?: unknown; | |
| 34 | +}> {} | |
| 35 | + | |
| 36 | +/** | |
| 37 | + * One scope's source store. `commit` writes a new snapshot from a full file set | |
| 38 | + * and returns its id. `read` materializes a snapshot back to a file set. | |
| 39 | + * `latest` is the most recent snapshot on the default branch (null when the | |
| 40 | + * scope has never published). `log` lists snapshots newest-first. | |
| 41 | + */ | |
| 42 | +export interface ScopeArtifactStore { | |
| 43 | + readonly commit: ( | |
| 44 | + files: FileSet, | |
| 45 | + message: string, | |
| 46 | + ) => Effect.Effect<SnapshotMeta, ArtifactStoreError>; | |
| 47 | + readonly read: (id: SnapshotId) => Effect.Effect<FileSet, ArtifactStoreError>; | |
| 48 | + readonly readFile: ( | |
| 49 | + id: SnapshotId, | |
| 50 | + path: string, | |
| 51 | + ) => Effect.Effect<string | null, ArtifactStoreError>; | |
| 52 | + readonly list: (id: SnapshotId) => Effect.Effect<readonly string[], ArtifactStoreError>; | |
| 53 | + readonly latest: () => Effect.Effect<SnapshotMeta | null, ArtifactStoreError>; | |
| 54 | + readonly log: (limit?: number) => Effect.Effect<readonly SnapshotMeta[], ArtifactStoreError>; | |
| 55 | +} | |
| 56 | + | |
| 57 | +/** The substrate-neutral store: hands out a per-scope store by scope key. */ | |
| 58 | +export interface ArtifactStore { | |
| 59 | + readonly forScope: (scope: string) => Effect.Effect<ScopeArtifactStore, ArtifactStoreError>; | |
| 60 | +} | |
| @@ -0,0 +1,8 @@ | ||
| 1 | +// The five substrate-neutral seams. Self-hosted backings live under | |
| 2 | +// `src/backing/`; each seam has a conformance suite (`*.conformance.ts`) that a | |
| 3 | +// backing must pass, keeping a future Cloudflare backing honest. | |
| 4 | +export * from "./artifact-store"; | |
| 5 | +export * from "./scope-db"; | |
| 6 | +export * from "./tool-sandbox"; | |
| 7 | +export * from "./workflow-runner"; | |
| 8 | +export * from "./live-channel"; | |
| @@ -0,0 +1,36 @@ | ||
| 1 | +import type { Effect } from "effect"; | |
| 2 | +import { Data } from "effect"; | |
| 3 | + | |
| 4 | +// --------------------------------------------------------------------------- | |
| 5 | +// LiveChannel — invalidation-only pub/sub for live UI data. | |
| 6 | +// | |
| 7 | +// The protocol is invalidation, never rows: a scope-db write bumps a table's | |
| 8 | +// version counter and publishes `{scope, table, version}`; a subscribed widget | |
| 9 | +// refetches through its authed query path. No row data ever crosses the channel | |
| 10 | +// (MCP Apps forbids server push through the host, so live data rides this side | |
| 11 | +// channel; but even here we only ship invalidations). The self-hosted backing | |
| 12 | +// is an in-process emitter exposed as SSE on the Bun server; the cloud backing | |
| 13 | +// (future) is the DO/facet socket owner. | |
| 14 | +// --------------------------------------------------------------------------- | |
| 15 | + | |
| 16 | +export class LiveChannelError extends Data.TaggedError("LiveChannelError")<{ | |
| 17 | + readonly message: string; | |
| 18 | + readonly cause?: unknown; | |
| 19 | +}> {} | |
| 20 | + | |
| 21 | +export interface Invalidation { | |
| 22 | + readonly scope: string; | |
| 23 | + readonly table: string; | |
| 24 | + readonly version: number; | |
| 25 | +} | |
| 26 | + | |
| 27 | +export interface LiveChannel { | |
| 28 | + /** Publish an invalidation to every subscriber of the scope. */ | |
| 29 | + readonly publish: (event: Invalidation) => Effect.Effect<void, LiveChannelError>; | |
| 30 | + /** | |
| 31 | + * Subscribe to a scope's invalidations. The listener fires for each publish; | |
| 32 | + * the returned thunk unsubscribes. Delivery is best-effort and unordered | |
| 33 | + * beyond per-table version monotonicity. | |
| 34 | + */ | |
| 35 | + readonly subscribe: (scope: string, listener: (event: Invalidation) => void) => () => void; | |
| 36 | +} | |
| @@ -0,0 +1,58 @@ | ||
| 1 | +import type { Effect } from "effect"; | |
| 2 | +import { Data } from "effect"; | |
| 3 | + | |
| 4 | +// --------------------------------------------------------------------------- | |
| 5 | +// ScopeDb — the per-scope app database (the shared primitive). | |
| 6 | +// | |
| 7 | +// One SQL database per scope: tools write it, workflows schedule tools, UI | |
| 8 | +// reads it, skills route the agent to it. The self-hosted backing is one libSQL | |
| 9 | +// (SQLite) file per scope; the cloud backing (future) is a Durable Object | |
| 10 | +// facet. The seam carries a template-tag `sql` (the author-facing shape) plus | |
| 11 | +// per-table version counters: every write bumps the touched table's version so | |
| 12 | +// the LiveChannel can publish an invalidation and widgets refetch. | |
| 13 | +// | |
| 14 | +// Scope isolation is structural: `forScope(a)` and `forScope(b)` are distinct | |
| 15 | +// databases; there is no cross-scope query path. | |
| 16 | +// --------------------------------------------------------------------------- | |
| 17 | + | |
| 18 | +export class ScopeDbError extends Data.TaggedError("ScopeDbError")<{ | |
| 19 | + readonly message: string; | |
| 20 | + readonly cause?: unknown; | |
| 21 | +}> {} | |
| 22 | + | |
| 23 | +/** The author-facing db handle (matches `executor:app`'s `ScopeDb`): a tagged | |
| 24 | + * template returning rows. Also exposes the tables a statement touched so the | |
| 25 | + * runtime can bump version counters and publish invalidations. */ | |
| 26 | +export interface ScopeDbHandle { | |
| 27 | + readonly sql: <Row = Record<string, unknown>>( | |
| 28 | + strings: TemplateStringsArray, | |
| 29 | + ...values: unknown[] | |
| 30 | + ) => Effect.Effect<readonly Row[], ScopeDbError>; | |
| 31 | + /** Run a raw statement (used by the runtime for probes/migrations). */ | |
| 32 | + readonly exec: <Row = Record<string, unknown>>( | |
| 33 | + sql: string, | |
| 34 | + params?: readonly unknown[], | |
| 35 | + ) => Effect.Effect<readonly Row[], ScopeDbError>; | |
| 36 | + /** Current version counter for a table (0 if never written). */ | |
| 37 | + readonly tableVersion: (table: string) => Effect.Effect<number, ScopeDbError>; | |
| 38 | + /** Snapshot of every tracked table's version. */ | |
| 39 | + readonly versions: () => Effect.Effect<ReadonlyMap<string, number>, ScopeDbError>; | |
| 40 | +} | |
| 41 | + | |
| 42 | +/** A write that touched tables, with the versions after the bump. Emitted so | |
| 43 | + * the runtime can drive `LiveChannel.publish`. */ | |
| 44 | +export interface ScopeWriteEvent { | |
| 45 | + readonly scope: string; | |
| 46 | + readonly tables: readonly { readonly table: string; readonly version: number }[]; | |
| 47 | +} | |
| 48 | + | |
| 49 | +export interface ScopeDb { | |
| 50 | + readonly forScope: (scope: string) => Effect.Effect<ScopeDbHandle, ScopeDbError>; | |
| 51 | + /** | |
| 52 | + * Subscribe to writes across scopes (the runtime wires this into LiveChannel). | |
| 53 | + * Returns an unsubscribe thunk. Best-effort in-process; the cloud backing | |
| 54 | + * makes the storage owner the notifier. | |
| 55 | + */ | |
| 56 | + readonly onWrite: (listener: (event: ScopeWriteEvent) => void) => () => void; | |
| 57 | + readonly close: () => Effect.Effect<void, ScopeDbError>; | |
| 58 | +} | |
| @@ -0,0 +1,105 @@ | ||
| 1 | +import type { Effect } from "effect"; | |
| 2 | +import { Data } from "effect"; | |
| 3 | + | |
| 4 | +// --------------------------------------------------------------------------- | |
| 5 | +// ToolSandbox — the isolated substrate that runs published bundles. | |
| 6 | +// | |
| 7 | +// Two operations, both over a bundled JS string (esbuild output, platform | |
| 8 | +// modules external, zod inlined): | |
| 9 | +// | |
| 10 | +// collect(bundle): import the bundle with NOTHING bound. `define*` calls | |
| 11 | +// return descriptors; a collector shim gathers them and returns JSON. This | |
| 12 | +// is how the publish pipeline extracts the versioned descriptor from source | |
| 13 | +// without ever running effectful code. Run twice + byte-compare = the | |
| 14 | +// determinism gate (a bundle that reads Math.random / Date.now at the top | |
| 15 | +// level or in a describe path diverges and is rejected). | |
| 16 | +// | |
| 17 | +// invoke(bundle, request): run one artifact's handler. The handler receives | |
| 18 | +// pre-bound clients whose method calls cross OUT through a serializable | |
| 19 | +// bridge (`HandleBridge.call`). EVERYTHING crossing the boundary is | |
| 20 | +// serializable — the cloud version of this seam is an RPC, so the interface | |
| 21 | +// forbids passing functions or live objects across. Fan-out arrays | |
| 22 | +// (`connections("gmail")` -> client[]) are modeled as indexed handle roots. | |
| 23 | +// | |
| 24 | +// The self-hosted backing is QuickJS (packages/kernel/runtime-quickjs), whose | |
| 25 | +// `SandboxToolInvoker.invoke({path, args})` already matches the bridge shape. | |
| 26 | +// The cloud backing (future) is Worker Loaders. The Deno subprocess kernel is | |
| 27 | +// the harder-isolation escalation behind this same seam. | |
| 28 | +// --------------------------------------------------------------------------- | |
| 29 | + | |
| 30 | +export class ToolSandboxError extends Data.TaggedError("ToolSandboxError")<{ | |
| 31 | + readonly message: string; | |
| 32 | + readonly kind: "collect" | "invoke" | "timeout" | "network" | "nondeterministic" | "bundle"; | |
| 33 | + readonly cause?: unknown; | |
| 34 | +}> {} | |
| 35 | + | |
| 36 | +/** | |
| 37 | + * The serializable bridge the sandbox calls out through. `root` names an | |
| 38 | + * injected handle (a connection role, `db`, or one element of a fan-out set); | |
| 39 | + * `path` is the method chain (`["events", "list"]`); `args` is the JSON call | |
| 40 | + * arguments. The return value is JSON. This is the ONE way sandboxed code | |
| 41 | + * reaches the host — nothing else is wired. | |
| 42 | + */ | |
| 43 | +export interface HandleBridge { | |
| 44 | + readonly call: (input: { | |
| 45 | + readonly root: string; | |
| 46 | + readonly path: readonly string[]; | |
| 47 | + readonly args: readonly unknown[]; | |
| 48 | + }) => Effect.Effect<unknown, ToolSandboxError>; | |
| 49 | +} | |
| 50 | + | |
| 51 | +/** Which handle roots to inject and, for fan-out roots, how many elements. A | |
| 52 | + * single connection role is `{ kind: "single" }`; a fan-out is | |
| 53 | + * `{ kind: "array", count }`; `db` is a single. Everything the handler can see | |
| 54 | + * is enumerated here — undeclared roots are simply absent. */ | |
| 55 | +export type HandleRootSpec = | |
| 56 | + | { readonly kind: "single" } | |
| 57 | + | { readonly kind: "array"; readonly count: number }; | |
| 58 | + | |
| 59 | +export interface InvokeRequest { | |
| 60 | + /** The artifact whose handler to run (path identity, e.g. `issues-sync`). */ | |
| 61 | + readonly artifact: string; | |
| 62 | + /** The kind selects the wrapper the sandbox uses to reach the handler. */ | |
| 63 | + readonly kind: "tool"; | |
| 64 | + /** JSON input passed to the handler. */ | |
| 65 | + readonly input: unknown; | |
| 66 | + /** The handle roots to inject, keyed by the name the handler destructures | |
| 67 | + * (`github`, `db`, `inboxes`). */ | |
| 68 | + readonly roots: Readonly<Record<string, HandleRootSpec>>; | |
| 69 | +} | |
| 70 | + | |
| 71 | +export interface InvokeResult { | |
| 72 | + readonly output: unknown; | |
| 73 | + readonly logs: readonly string[]; | |
| 74 | +} | |
| 75 | + | |
| 76 | +/** A collected artifact descriptor — the JSON `define*` returns. The pipeline | |
| 77 | + * refines this into the versioned descriptor; the sandbox only guarantees it | |
| 78 | + * is deterministic JSON. */ | |
| 79 | +export interface CollectedArtifact { | |
| 80 | + readonly kind: "tool" | "workflow"; | |
| 81 | + readonly descriptor: unknown; | |
| 82 | +} | |
| 83 | + | |
| 84 | +export interface CollectResult { | |
| 85 | + /** Descriptors keyed by artifact path identity. */ | |
| 86 | + readonly artifacts: Readonly<Record<string, CollectedArtifact>>; | |
| 87 | +} | |
| 88 | + | |
| 89 | +export interface ToolSandbox { | |
| 90 | + /** | |
| 91 | + * Import the bundle with nothing bound and gather every `define*` descriptor. | |
| 92 | + * Runs the collection twice internally and byte-compares; a mismatch fails | |
| 93 | + * with `kind: "nondeterministic"`. This is the determinism gate. | |
| 94 | + */ | |
| 95 | + readonly collect: (bundle: string) => Effect.Effect<CollectResult, ToolSandboxError>; | |
| 96 | + /** | |
| 97 | + * Run one artifact's handler with injected handles bridged through `bridge`. | |
| 98 | + * Network is denied; a per-call timeout kills a runaway handler. | |
| 99 | + */ | |
| 100 | + readonly invoke: ( | |
| 101 | + bundle: string, | |
| 102 | + request: InvokeRequest, | |
| 103 | + bridge: HandleBridge, | |
| 104 | + ) => Effect.Effect<InvokeResult, ToolSandboxError>; | |
| 105 | +} | |
| @@ -0,0 +1,139 @@ | ||
| 1 | +import type { Effect } from "effect"; | |
| 2 | +import { Data } from "effect"; | |
| 3 | + | |
| 4 | +// --------------------------------------------------------------------------- | |
| 5 | +// WorkflowRunner — durable workflow execution with an event-sourced journal. | |
| 6 | +// | |
| 7 | +// Cloudflare Workflows semantics verbatim: `step.do(name, fn)` memoized by the | |
| 8 | +// journal, `step.sleep`, `step.waitForEvent`, plus executor's `step.tool` | |
| 9 | +// (journal + audit in one). Kill mid-step -> restart -> completed steps do NOT | |
| 10 | +// re-execute (replay reads the journal). The self-hosted backing is a SQLite | |
| 11 | +// journal replay runner + in-process scheduler; the cloud backing (future) is | |
| 12 | +// CF Workflows + dynamic-workflows. The journal shape is modeled on | |
| 13 | +// vercel/workflow's World Storage contract (append-only events, materialized | |
| 14 | +// run/step views). | |
| 15 | +// | |
| 16 | +// The runner is substrate-neutral: it takes a `StepExecutor` (how to run a | |
| 17 | +// named step body / tool call for THIS run) so the sandbox/tool wiring stays | |
| 18 | +// outside the durable core. Runs pin the snapshot that started them. | |
| 19 | +// --------------------------------------------------------------------------- | |
| 20 | + | |
| 21 | +export class WorkflowError extends Data.TaggedError("WorkflowError")<{ | |
| 22 | + readonly message: string; | |
| 23 | + readonly cause?: unknown; | |
| 24 | +}> {} | |
| 25 | + | |
| 26 | +export type RunStatus = "running" | "sleeping" | "waiting" | "completed" | "failed" | "cancelled"; | |
| 27 | + | |
| 28 | +export interface RunView { | |
| 29 | + readonly runId: string; | |
| 30 | + readonly scope: string; | |
| 31 | + readonly workflow: string; | |
| 32 | + readonly snapshotId: string; | |
| 33 | + readonly status: RunStatus; | |
| 34 | + readonly input: unknown; | |
| 35 | + readonly output?: unknown; | |
| 36 | + readonly error?: string; | |
| 37 | + readonly createdAt: number; | |
| 38 | + readonly updatedAt: number; | |
| 39 | +} | |
| 40 | + | |
| 41 | +export type StepStatus = "completed" | "failed"; | |
| 42 | + | |
| 43 | +export interface StepView { | |
| 44 | + readonly runId: string; | |
| 45 | + readonly name: string; | |
| 46 | + readonly status: StepStatus; | |
| 47 | + readonly output?: unknown; | |
| 48 | + readonly error?: string; | |
| 49 | + readonly attempt: number; | |
| 50 | + readonly completedAt: number; | |
| 51 | +} | |
| 52 | + | |
| 53 | +/** How the runner executes step bodies for a specific run. The durable core | |
| 54 | + * drives this; the caller supplies it (bound to the run's snapshot + scope + | |
| 55 | + * the tool invoke path). Each method is only ever called for a step that has | |
| 56 | + * NOT completed in the journal (the runner checks the journal first). */ | |
| 57 | +export interface StepExecutor { | |
| 58 | + /** Run a `step.do` body identified by name; return its JSON result. */ | |
| 59 | + readonly runStep: (name: string) => Effect.Effect<unknown, WorkflowError>; | |
| 60 | + /** Run a `step.tool` call (address + args); return the tool's JSON result. | |
| 61 | + * Journaled AND audited as a tool run by the caller's implementation. */ | |
| 62 | + readonly runTool: ( | |
| 63 | + name: string, | |
| 64 | + address: string, | |
| 65 | + args: unknown, | |
| 66 | + ) => Effect.Effect<unknown, WorkflowError>; | |
| 67 | + /** Deliver a `step.notify`. */ | |
| 68 | + readonly notify: (msg: { | |
| 69 | + readonly title: string; | |
| 70 | + readonly body?: string; | |
| 71 | + readonly link?: string; | |
| 72 | + }) => Effect.Effect<void, WorkflowError>; | |
| 73 | +} | |
| 74 | + | |
| 75 | +/** The workflow body, expressed against the durable step API. The runner | |
| 76 | + * replays it: completed steps resolve from the journal, the first incomplete | |
| 77 | + * step actually executes. `sleep`/`waitForEvent` suspend the run. */ | |
| 78 | +export interface DurableSteps { | |
| 79 | + readonly do: <T>(name: string, fn: () => Promise<T> | T) => Promise<T>; | |
| 80 | + readonly tool: <T = unknown>(address: string, args: Record<string, unknown>) => Promise<T>; | |
| 81 | + readonly sleep: (name: string, ms: number) => Promise<void>; | |
| 82 | + readonly waitForEvent: <T = unknown>(name: string, opts?: { timeout?: number }) => Promise<T>; | |
| 83 | + readonly notify: (msg: { title: string; body?: string; link?: string }) => Promise<void>; | |
| 84 | +} | |
| 85 | + | |
| 86 | +export interface StartRunInput { | |
| 87 | + readonly scope: string; | |
| 88 | + readonly workflow: string; | |
| 89 | + readonly snapshotId: string; | |
| 90 | + readonly input: unknown; | |
| 91 | + /** Optional caller-supplied run id (idempotent starts / scheduler dedupe). */ | |
| 92 | + readonly runId?: string; | |
| 93 | +} | |
| 94 | + | |
| 95 | +/** | |
| 96 | + * The durable runner. `start` creates a run and drives it to its first | |
| 97 | + * suspension or completion. `resume` re-drives a suspended/interrupted run from | |
| 98 | + * the journal (this is what makes the kill test pass: completed steps replay | |
| 99 | + * from the journal and never re-execute). `signal` delivers a waitForEvent | |
| 100 | + * payload. `get`/`listSteps` are the queryable status/history. | |
| 101 | + * | |
| 102 | + * `run` is the workflow body supplied by the caller (bound to the snapshot's | |
| 103 | + * compiled bundle via `execute`). The runner calls it with a `DurableSteps` | |
| 104 | + * that consults the journal. | |
| 105 | + */ | |
| 106 | +export interface WorkflowRunner { | |
| 107 | + readonly start: ( | |
| 108 | + input: StartRunInput, | |
| 109 | + execute: (steps: DurableSteps) => Promise<unknown>, | |
| 110 | + ) => Effect.Effect<RunView, WorkflowError>; | |
| 111 | + readonly resume: ( | |
| 112 | + runId: string, | |
| 113 | + execute: (steps: DurableSteps) => Promise<unknown>, | |
| 114 | + ) => Effect.Effect<RunView, WorkflowError>; | |
| 115 | + readonly signal: ( | |
| 116 | + runId: string, | |
| 117 | + eventName: string, | |
| 118 | + payload: unknown, | |
| 119 | + execute: (steps: DurableSteps) => Promise<unknown>, | |
| 120 | + ) => Effect.Effect<RunView, WorkflowError>; | |
| 121 | + readonly cancel: (runId: string) => Effect.Effect<RunView, WorkflowError>; | |
| 122 | + readonly get: (runId: string) => Effect.Effect<RunView | null, WorkflowError>; | |
| 123 | + readonly list: (filter?: { | |
| 124 | + readonly scope?: string; | |
| 125 | + readonly workflow?: string; | |
| 126 | + }) => Effect.Effect<readonly RunView[], WorkflowError>; | |
| 127 | + readonly listSteps: (runId: string) => Effect.Effect<readonly StepView[], WorkflowError>; | |
| 128 | + readonly close: () => Effect.Effect<void, WorkflowError>; | |
| 129 | +} | |
| 130 | + | |
| 131 | +/** Control errors that steer retry/replay, matching CF's vocabulary. */ | |
| 132 | +export class FatalError extends Data.TaggedError("FatalError")<{ | |
| 133 | + readonly message: string; | |
| 134 | +}> {} | |
| 135 | + | |
| 136 | +export class RetryableError extends Data.TaggedError("RetryableError")<{ | |
| 137 | + readonly message: string; | |
| 138 | + readonly retryAfter?: number; | |
| 139 | +}> {} | |
| @@ -0,0 +1,197 @@ | ||
| 1 | +// --------------------------------------------------------------------------- | |
| 2 | +// The daily-brief fixture set, as an in-memory file set (path -> contents). | |
| 3 | +// | |
| 4 | +// Adapted from prototypes/executor-artifacts to be executable against the | |
| 5 | +// implemented contract: `tools/issues-sync.ts` writes the scope `issues` table, | |
| 6 | +// `workflows/morning-sync.ts` schedules it and flags stale issues, | |
| 7 | +// `ui/dashboard.tsx` reads the table, `skills/issues-brief/SKILL.md` routes the | |
| 8 | +// agent. These are payloads (author code), not package source, so they live as | |
| 9 | +// strings rather than compiled files. | |
| 10 | +// --------------------------------------------------------------------------- | |
| 11 | + | |
| 12 | +export const DAILY_BRIEF_MANIFEST = `{ | |
| 13 | + "$schema": "https://executor.sh/schemas/scope-manifest.json", | |
| 14 | + "scope": "rhys", | |
| 15 | + "description": "Personal scope artifacts — GitHub issues brief.", | |
| 16 | + "connections": { "github": "github/rhys" }, | |
| 17 | + "artifacts": { "skills": "skills/", "tools": "tools/", "workflows": "workflows/", "ui": "ui/" } | |
| 18 | +} | |
| 19 | +`; | |
| 20 | + | |
| 21 | +export const ISSUES_SYNC_TS = `import { z } from "zod"; | |
| 22 | +import { defineTool, connection } from "executor:app"; | |
| 23 | + | |
| 24 | +export default defineTool({ | |
| 25 | + description: | |
| 26 | + "Refresh the scope \\\`issues\\\` table from GitHub. Syncs open issues across the given repos (default: every repo the connection can see).", | |
| 27 | + connections: { | |
| 28 | + github: connection("github", { description: "GitHub account whose issues to sync" }), | |
| 29 | + }, | |
| 30 | + input: z.object({ | |
| 31 | + repos: z.array(z.string()).optional().describe("owner/repo entries; omit to sync all accessible repos"), | |
| 32 | + since: z.string().optional().describe("ISO timestamp — only issues updated after this"), | |
| 33 | + }), | |
| 34 | + output: z.object({ synced: z.number(), repos: z.number() }), | |
| 35 | + annotations: { readOnly: false, destructive: false }, | |
| 36 | + async handler({ repos, since }, { github, db }) { | |
| 37 | + const targets = | |
| 38 | + repos ?? | |
| 39 | + (await github.repos.listForAuthenticatedUser({ per_page: 100 })).map((r) => r.full_name); | |
| 40 | + | |
| 41 | + await db.sql\` | |
| 42 | + CREATE TABLE IF NOT EXISTS issues ( | |
| 43 | + repo TEXT NOT NULL, number INTEGER NOT NULL, title TEXT NOT NULL, | |
| 44 | + labels TEXT NOT NULL DEFAULT '[]', assignee TEXT, updated_at TEXT NOT NULL, url TEXT NOT NULL, | |
| 45 | + PRIMARY KEY (repo, number) | |
| 46 | + )\`; | |
| 47 | + | |
| 48 | + let synced = 0; | |
| 49 | + for (const target of targets) { | |
| 50 | + const [owner, repo] = target.split("/"); | |
| 51 | + const issues = await github.issues.listForRepo({ owner, repo, state: "open", since, per_page: 100 }); | |
| 52 | + for (const issue of issues) { | |
| 53 | + if (issue.pull_request) continue; | |
| 54 | + await db.sql\` | |
| 55 | + INSERT INTO issues (repo, number, title, labels, assignee, updated_at, url) | |
| 56 | + VALUES (\${target}, \${issue.number}, \${issue.title}, | |
| 57 | + \${JSON.stringify(issue.labels.map((l) => l.name))}, | |
| 58 | + \${issue.assignee?.login ?? null}, \${issue.updated_at}, \${issue.html_url}) | |
| 59 | + ON CONFLICT (repo, number) DO UPDATE SET | |
| 60 | + title = excluded.title, labels = excluded.labels, | |
| 61 | + assignee = excluded.assignee, updated_at = excluded.updated_at\`; | |
| 62 | + synced++; | |
| 63 | + } | |
| 64 | + } | |
| 65 | + return { synced, repos: targets.length }; | |
| 66 | + }, | |
| 67 | +}); | |
| 68 | +`; | |
| 69 | + | |
| 70 | +export const SEARCH_ALL_MAIL_TS = `import { z } from "zod"; | |
| 71 | +import { defineTool, connections } from "executor:app"; | |
| 72 | + | |
| 73 | +export default defineTool({ | |
| 74 | + description: | |
| 75 | + "Search across all connected Gmail accounts. Returns matches newest-first, tagged with which inbox they came from.", | |
| 76 | + connections: { | |
| 77 | + inboxes: connections("gmail", { description: "Gmail accounts to search across" }), | |
| 78 | + }, | |
| 79 | + input: z.object({ | |
| 80 | + query: z.string().describe("Gmail search syntax, e.g. from:acme subject:invoice"), | |
| 81 | + limit: z.number().int().min(1).max(100).default(25), | |
| 82 | + }), | |
| 83 | + output: z.object({ | |
| 84 | + results: z.array(z.object({ | |
| 85 | + inbox: z.string(), id: z.string(), from: z.string(), | |
| 86 | + subject: z.string(), snippet: z.string(), date: z.string(), | |
| 87 | + })), | |
| 88 | + }), | |
| 89 | + annotations: { readOnly: true }, | |
| 90 | + async handler({ query, limit }, { inboxes }) { | |
| 91 | + const perInbox = await Promise.all( | |
| 92 | + inboxes.map(async (inbox) => { | |
| 93 | + const { messages } = await inbox.messages.search({ q: query, maxResults: limit }); | |
| 94 | + return messages.map((m) => ({ | |
| 95 | + inbox: inbox.account.email, id: m.id, from: m.from, | |
| 96 | + subject: m.subject, snippet: m.snippet, date: m.date, | |
| 97 | + })); | |
| 98 | + }), | |
| 99 | + ); | |
| 100 | + const results = perInbox.flat().sort((a, b) => b.date.localeCompare(a.date)).slice(0, limit); | |
| 101 | + return { results }; | |
| 102 | + }, | |
| 103 | +}); | |
| 104 | +`; | |
| 105 | + | |
| 106 | +export const MORNING_SYNC_TS = `import { defineWorkflow } from "executor:app"; | |
| 107 | + | |
| 108 | +export default defineWorkflow({ | |
| 109 | + description: "Refresh GitHub issues at 9am and flag stale ones.", | |
| 110 | + schedule: { cron: "0 9 * * 1-5", timezone: "America/New_York" }, | |
| 111 | + async run(step, { db }) { | |
| 112 | + const { synced } = await step.tool("issues-sync", {}); | |
| 113 | + const stale = await step.do("find-stale", () => | |
| 114 | + db.sql\` | |
| 115 | + SELECT repo, number, title FROM issues | |
| 116 | + WHERE updated_at < datetime('now', '-14 days') | |
| 117 | + ORDER BY updated_at ASC\`, | |
| 118 | + ); | |
| 119 | + if (stale.length > 0) { | |
| 120 | + await step.notify({ | |
| 121 | + title: \`\${stale.length} issues stale >14 days\`, | |
| 122 | + body: stale.slice(0, 5).map((i) => \`\${i.repo}#\${i.number} — \${i.title}\`).join("\\n"), | |
| 123 | + link: "ui://dashboard", | |
| 124 | + }); | |
| 125 | + } | |
| 126 | + return { synced, stale: stale.length }; | |
| 127 | + }, | |
| 128 | +}); | |
| 129 | +`; | |
| 130 | + | |
| 131 | +export const DASHBOARD_TSX = `import { useQuery, useTool, config } from "executor:ui"; | |
| 132 | +import { Card, CardContent, CardHeader, CardTitle, Badge, Button, Input } from "executor:ui/components"; | |
| 133 | +import { useState } from "react"; | |
| 134 | + | |
| 135 | +config({ maxHeight: 720, title: "GitHub Issues" }); | |
| 136 | + | |
| 137 | +export default function App() { | |
| 138 | + const [filter, setFilter] = useState(""); | |
| 139 | + const issues = useQuery((db) => db.sql\`SELECT * FROM issues ORDER BY updated_at DESC\`); | |
| 140 | + const sync = useTool("issues-sync"); | |
| 141 | + if (issues.isLoading) return <Card><CardContent>Loading issues…</CardContent></Card>; | |
| 142 | + const rows = issues.data.filter( | |
| 143 | + (i) => !filter || i.repo.includes(filter) || i.title.toLowerCase().includes(filter.toLowerCase()), | |
| 144 | + ); | |
| 145 | + return ( | |
| 146 | + <div className="flex flex-col gap-4 p-1"> | |
| 147 | + <div className="flex items-center gap-2"> | |
| 148 | + <Input placeholder="Filter…" value={filter} onChange={(e) => setFilter(e.target.value)} /> | |
| 149 | + <Button disabled={sync.isRunning} onClick={() => sync.run({}).then(() => issues.refetch())}> | |
| 150 | + {sync.isRunning ? "Syncing…" : "Sync now"} | |
| 151 | + </Button> | |
| 152 | + <span className="ml-auto">{rows.length} open issues</span> | |
| 153 | + </div> | |
| 154 | + <Card> | |
| 155 | + <CardHeader><CardTitle>Open issues</CardTitle></CardHeader> | |
| 156 | + <CardContent> | |
| 157 | + {rows.slice(0, 30).map((issue) => ( | |
| 158 | + <a key={\`\${issue.repo}#\${issue.number}\`} href={issue.url}> | |
| 159 | + <span>{issue.repo}#{issue.number}</span> <span>{issue.title}</span> | |
| 160 | + {JSON.parse(issue.labels).map((l) => (<Badge key={l}>{l}</Badge>))} | |
| 161 | + </a> | |
| 162 | + ))} | |
| 163 | + </CardContent> | |
| 164 | + </Card> | |
| 165 | + </div> | |
| 166 | + ); | |
| 167 | +} | |
| 168 | +`; | |
| 169 | + | |
| 170 | +export const ISSUES_BRIEF_SKILL = `--- | |
| 171 | +name: issues-brief | |
| 172 | +description: Answer questions about open GitHub issues from the scope \`issues\` database instead of paging the GitHub API; sync first if stale. Use when the user asks about their issues, the issue backlog, stale issues, or anything GitHub-issue related. | |
| 173 | +--- | |
| 174 | + | |
| 175 | +# GitHub issues brief | |
| 176 | + | |
| 177 | +The scope database has an \`issues\` table maintained by the \`issues-sync\` | |
| 178 | +tool and refreshed weekday mornings by the \`morning-sync\` workflow. Answer | |
| 179 | +issue questions from this table — do not page the GitHub API directly. | |
| 180 | + | |
| 181 | +## Recipe | |
| 182 | + | |
| 183 | +1. Check freshness: \`SELECT MAX(updated_at) FROM issues\`. If older than ~24h, call \`issues-sync\` first. | |
| 184 | +2. Query the table for what was asked. | |
| 185 | +3. For anything visual, open the authored dashboard: \`ui://dashboard\`. | |
| 186 | +`; | |
| 187 | + | |
| 188 | +/** The daily-brief file set as a path -> contents map. */ | |
| 189 | +export const dailyBriefFileSet = (): Map<string, string> => | |
| 190 | + new Map<string, string>([ | |
| 191 | + ["executor.json", DAILY_BRIEF_MANIFEST], | |
| 192 | + ["tools/issues-sync.ts", ISSUES_SYNC_TS], | |
| 193 | + ["tools/search-all-mail.ts", SEARCH_ALL_MAIL_TS], | |
| 194 | + ["workflows/morning-sync.ts", MORNING_SYNC_TS], | |
| 195 | + ["ui/dashboard.tsx", DASHBOARD_TSX], | |
| 196 | + ["skills/issues-brief/SKILL.md", ISSUES_BRIEF_SKILL], | |
| 197 | + ]); | |
| @@ -0,0 +1,23 @@ | ||
| 1 | +{ | |
| 2 | + "compilerOptions": { | |
| 3 | + "target": "ES2022", | |
| 4 | + "module": "ESNext", | |
| 5 | + "moduleResolution": "Bundler", | |
| 6 | + "strict": true, | |
| 7 | + "skipLibCheck": true, | |
| 8 | + "lib": ["ES2022", "DOM"], | |
| 9 | + "types": ["bun-types", "node"], | |
| 10 | + "noUnusedLocals": true, | |
| 11 | + "noImplicitOverride": true, | |
| 12 | + "jsx": "react-jsx", | |
| 13 | + "plugins": [ | |
| 14 | + { | |
| 15 | + "name": "@effect/language-service", | |
| 16 | + "ignoreEffectSuggestionsInTscExitCode": true, | |
| 17 | + "ignoreEffectWarningsInTscExitCode": true, | |
| 18 | + "diagnosticSeverity": {} | |
| 19 | + } | |
| 20 | + ] | |
| 21 | + }, | |
| 22 | + "include": ["src/**/*.ts", "src/**/*.tsx"] | |
| 23 | +} | |
| @@ -0,0 +1,15 @@ | ||
| 1 | +import { defineConfig } from "tsup"; | |
| 2 | + | |
| 3 | +export default defineConfig({ | |
| 4 | + entry: { | |
| 5 | + index: "src/index.ts", | |
| 6 | + api: "src/api.ts", | |
| 7 | + "seams/index": "src/seams/index.ts", | |
| 8 | + "testing/index": "src/testing/index.ts", | |
| 9 | + }, | |
| 10 | + format: ["esm"], | |
| 11 | + dts: false, | |
| 12 | + sourcemap: true, | |
| 13 | + clean: true, | |
| 14 | + external: [/^@executor-js\//, /^effect/, /^@effect\//, /^@modelcontextprotocol\//, "esbuild"], | |
| 15 | +}); | |
| @@ -0,0 +1,9 @@ | ||
| 1 | +import { defineConfig } from "vitest/config"; | |
| 2 | + | |
| 3 | +export default defineConfig({ | |
| 4 | + test: { | |
| 5 | + include: ["src/**/*.test.ts"], | |
| 6 | + testTimeout: 30_000, | |
| 7 | + hookTimeout: 30_000, | |
| 8 | + }, | |
| 9 | +}); | |