Every span we emit starts inside the fetch handler, so nothing could tell a cold isolate paying Start's loadEntries import from an isolate that came up slowly before our code ran. Measured with a temporary entry-lag probe against wrangler tail event timestamps: a cold isolate spends 840-1844ms before our first line, a reused one 0-2ms. Stamp request_seq and start_graph.entered on the worker spans so cold-vs-warm is queryable from Axiom without a probe. Also drops a stray plans file that #1692 swept in via git add -A.
Unified · Split
| @@ -103,12 +103,26 @@ export { McpExecutionOwnerDirectoryDO } from "@executor-js/cloudflare/mcp/execut · 102 unmodified lines | ||
| 103 | 103 | // until the in-flight export resolves. |
| 104 | 104 | // --------------------------------------------------------------------------- |
| 105 | 105 | |
| 106 | −const fetchHandler = handler.fetch as ( | |
| 106 | +const rawFetchHandler = handler.fetch as ( | |
| 107 | 107 | request: Request, |
| 108 | 108 | env: Env, |
| 109 | 109 | ctx: ExecutionContext, |
| 110 | 110 | ) => Response | Promise<Response>; |
| 111 | 111 | |
| 112 | +/** | |
| 113 | + * Every entry into TanStack Start goes through here so `startGraphEntered` | |
| 114 | + * reflects whether this isolate has already paid the lazy `loadEntries` | |
| 115 | + * import — the cost that dominates a cold page request. | |
| 116 | + */ | |
| 117 | +const fetchHandler = ( | |
| 118 | + request: Request, | |
| 119 | + env: Env, | |
| 120 | + ctx: ExecutionContext, | |
| 121 | +): Response | Promise<Response> => { | |
| 122 | + markStartGraphEntered(); | |
| 123 | + return rawFetchHandler(request, env, ctx); | |
| 124 | +}; | |
| 125 | + | |
| 112 | 126 | const tracer = trace.getTracer("executor-cloud-worker"); |
| 113 | 127 | |
| 114 | 128 | const traceparentValueFor = (spanContext: SpanContext): string => |
| @@ -178,8 +192,43 @@ const mcpAgentHandler = makeCloudMcpAgentHandler({ · 63 unmodified lines | ||
| 178 | 192 | traceRequest: traceCloudMcpRequest, |
| 179 | 193 | }); |
| 180 | 194 | |
| 195 | +// --------------------------------------------------------------------------- | |
| 196 | +// Isolate lifecycle signals | |
| 197 | +// --------------------------------------------------------------------------- | |
| 198 | +// | |
| 199 | +// The Aug 2026 page-latency hunt kept stalling on one blind spot: every span we | |
| 200 | +// emit starts INSIDE the fetch handler, so nothing could distinguish "this | |
| 201 | +// isolate is cold and paying Start's lazy `loadEntries` import" from "this | |
| 202 | +// isolate came up slowly before our code ran at all". Both look like one slow | |
| 203 | +// span. These three attributes make the distinction queryable: | |
| 204 | +// | |
| 205 | +// executor.isolate.request_seq - 1 means this request is the isolate's | |
| 206 | +// first; page requests measured 1.04 per | |
| 207 | +// isolate, which is why nearly every one | |
| 208 | +// pays the cold cost. | |
| 209 | +// executor.start_graph.entered - whether anything had already driven | |
| 210 | +// TanStack Start in this isolate. /mcp and | |
| 211 | +// the passthrough proxies return before | |
| 212 | +// `fetchHandler`, so an isolate can serve | |
| 213 | +// plenty of traffic and still be cold here. | |
| 214 | +// Measured with a temporary entry-lag probe correlated against `wrangler tail` | |
| 215 | +// event timestamps: a cold isolate (`request_seq` 1) spends **840-1844ms** | |
| 216 | +// coming up before our first line runs, while a reused one spends **0-2ms**. | |
| 217 | +// That startup is invisible to every span we emit, and it sits on top of the | |
| 218 | +// ~3.1s `loadEntries` import — together the 3-5s a signed-in page costs. | |
| 219 | +// | |
| 220 | +// Both counters are cheap: two increments and no I/O. | |
| 221 | +let isolateRequestSeq = 0; | |
| 222 | +let startGraphEntered = false; | |
| 223 | + | |
| 224 | +const markStartGraphEntered = (): void => { | |
| 225 | + startGraphEntered = true; | |
| 226 | +}; | |
| 227 | + | |
| 181 | 228 | const cloudflareHandler: ExportedHandler<Env> = { |
| 182 | 229 | fetch: async (request, env, ctx) => { |
| 230 | + isolateRequestSeq += 1; | |
| 231 | + | |
| 183 | 232 | // Public pages must not enter TanStack Start: its first-request dynamic |
| 184 | 233 | // import loads the entire React + Effect server graph and can take seconds |
| 185 | 234 | // on a cold isolate. Classify and service-bind marketing at the Worker |
| @@ -250,6 +299,8 @@ const cloudflareHandler: ExportedHandler<Env> = { · 64 unmodified lines | ||
| 250 | 299 | async (span) => { |
| 251 | 300 | span.setAttribute(ATTR_HTTP_REQUEST_METHOD, request.method); |
| 252 | 301 | span.setAttribute(ATTR_URL_PATH, url.pathname); |
| 302 | + span.setAttribute("executor.isolate.request_seq", isolateRequestSeq); | |
| 303 | + span.setAttribute("executor.start_graph.entered", startGraphEntered); | |
| 253 | 304 | // oxlint-disable-next-line executor/no-try-catch-or-throw -- adapter boundary; observe response/error for span status, keep the flush alive past the response |
| 254 | 305 | try { |
| 255 | 306 | const response = await fetchHandler( |
| @@ -1,515 +0,0 @@ | ||
| 1 | −# Kill the plugin system — in-place migration plan | |
| 2 | − | |
| 3 | −Decision (Rhys, 2026-07-27): abandon the big-bang rework | |
| 4 | −(`UsefulSoftwareCo/executor-rework`, cloned read-only at | |
| 5 | −`.reference/executor-rework`); fix this repo in place over a sequence of PRs, | |
| 6 | −each keeping format/lint/typecheck/test/e2e green. Priority order: **data | |
| 7 | −models first**, then the plugin kill, then the concept-package (apps) model, | |
| 8 | −then the minimal SDK / platform split. | |
| 9 | − | |
| 10 | −RESEQUENCED (Rhys, 2026-07-28): **Phase 3 (apps/concept-package model) now | |
| 11 | −lands BEFORE Phase 2 (plugin kill), side by side with the protocol plugins.** | |
| 12 | −Rationale: the new model's tables are greenfield and additive — landing it | |
| 13 | −first means the legacy apps tables never need an in-place migration; the whole | |
| 14 | −model swaps underneath while openapi/mcp/graphql keep running as plugins. | |
| 15 | −Structurally sound because (a) the kernel + MCP host are already plugin-free, | |
| 16 | −so the new model's runtime never touched the registry, and (b) the apps plugin | |
| 17 | −already exercises every extension point, so the new model ships as the apps | |
| 18 | −plugin's new internals wearing the same costume. When 2.3 later converts the | |
| 19 | −four kinds, apps converts as a stable, proven model instead of mid-rewrite. | |
| 20 | −Phase 0 and 2.1 remain freely parallel — they don't touch apps. | |
| 21 | − | |
| 22 | −Hard line: side-by-side means new-apps-model alongside the _protocol plugins_, | |
| 23 | −never two app models with dual read paths inside the apps kind. Existing app | |
| 24 | −sources get a one-off migration to published-source form (enumerated | |
| 25 | −invariants, explicit approval, no permanent compatibility path). Accepted | |
| 26 | −cost: the new model's HTTP surface temporarily wears plugin-route ceremony | |
| 27 | −(HttpApiGroup + handler layers) that 2.2/2.6 later deletes — wiring, not | |
| 28 | −model. The authoring SDK (`defineApp`/`defineProvider`/`defineTool`) goes | |
| 29 | −public early; its boundaries take Standard Schema (StandardSchemaV1 & | |
| 30 | −StandardJSONSchemaV1), never Effect Schema, and Phase 4's vocabulary rules | |
| 31 | −(opaque owner refs, no "tenant") bind on it from day one. | |
| 32 | − | |
| 33 | −## Diagnosis (verified against the code) | |
| 34 | − | |
| 35 | −**The plugin system is a fiction.** `packages/core/sdk/src/plugin.ts` (~850 | |
| 36 | −lines, ~20 optional hooks) pretends the plugin set is open, but: | |
| 37 | − | |
| 38 | −- `Integration.kind` _is_ the plugin id string; all hot paths dispatch via | |
| 39 | − `runtimes.get(row.plugin_id)` (`executor.ts`). | |
| 40 | −- Core and React hardcode the closed set anyway: `migration-spec.ts` hardcodes | |
| 41 | − plugin ids; `packages/react` has `KIND_TO_PLUGIN_KEY` maps and an | |
| 42 | − `if (pluginId === "toolkits")` route branch. | |
| 43 | −- Nobody ever shipped a fourth protocol plugin, and the two provider plugins | |
| 44 | − (google/microsoft) were already collapsed into openapi (#1366). | |
| 45 | −- Whole packages exist only to support the open set: `plugin-routes.ts` | |
| 46 | − type-union machinery, `@executor-js/config`'s jsonc+jiti loader, | |
| 47 | − `packages/core/vite-plugin`'s virtual client module. The jsonc plugin path is | |
| 48 | − vestigial — no `executor.jsonc` exists anywhere in the repo, no docs mention | |
| 49 | − it. | |
| 50 | −- Toolkits (~880 LoC of pure curation + the single `toolPolicyProvider` slot) | |
| 51 | − and apps (~6.8k custom-tools gateway) are first-party capabilities wearing a | |
| 52 | − plugin costume. Kernel packages and `packages/hosts/mcp` have zero plugin | |
| 53 | − references — the indirection is not load-bearing for the real surfaces. | |
| 54 | − | |
| 55 | −**The data model is connection-first, not user-first** | |
| 56 | −(`packages/core/sdk/src/core-schema.ts`): | |
| 57 | − | |
| 58 | −- No user/subject table. `subject` is an opaque partition-key string smeared | |
| 59 | − across owned tables; a user "exists" only if they have a connection row. | |
| 60 | −- Disconnect is a hard delete (`connectionsRemove`) — a signed-out user is | |
| 61 | − bit-identical to one who never connected. "Which users are signed out" is | |
| 62 | − structurally unanswerable. | |
| 63 | −- Every Executor is bound to one `{tenant, subject}` via `owner-policy.ts`; | |
| 64 | − there is no cross-subject read path, so no admin view. | |
| 65 | −- Toolkit state lives in generic `plugin_storage` KV, not core tables. | |
| 66 | −- Cloud's real identity tables (WorkOS accounts/organizations) have no join to | |
| 67 | − `subject`. | |
| 68 | − | |
| 69 | −**What to salvage from the rework** (its model docs are the valuable artifact; | |
| 70 | −its implementation ported only ~10 of 131 legacy e2e scenarios — that's why it | |
| 71 | −failed, and why this plan is incremental): | |
| 72 | − | |
| 73 | −- Apps-as-published-source: `defineApp`/`defineProvider`/`defineTool`/ | |
| 74 | − `defineToolGroup`, immutable content-hashed versions, manifest derived by | |
| 75 | − executing the source, rollback = pointer move. | |
| 76 | −- Generators: OpenAPI/MCP specs generate the _same_ app source a human would | |
| 77 | − write, with a re-verifiable `GeneratorClaim` (first-party vs third-party). | |
| 78 | −- Minimal SDK boundary: `createExecutor`/`createRemoteExecutor`, no Effect | |
| 79 | − crossing the boundary, MCP exposure returns a standard MCP SDK server. | |
| 80 | −- Custody separation: secrets never on the wire; opaque owner refs. | |
| 81 | − | |
| 82 | −## Constraints discovered (do not relearn these) | |
| 83 | − | |
| 84 | −1. **Published packages make this semver-major.** `scripts/publish-packages.ts` | |
| 85 | − publishes `@executor-js/{config,sdk}` and plugins | |
| 86 | − `{example,file-secrets,graphql,keychain,mcp,onepassword,openapi}` — killing | |
| 87 | − `definePlugin` is a breaking release for seven packages. Changesets + | |
| 88 | − deprecation story required; `publish-packages.ts`, `pkg-pr-new.yml`, | |
| 89 | − `check-changelog-stubs.ts`, and `examples/*` all have hardcoded lists. | |
| 90 | −2. **`plugin_id = 'google'` is a live persisted value with no runtime.** The | |
| 91 | − provider-service-split migration skips orgs with missing spec blobs | |
| 92 | − (`sqlite.test.ts` asserts survivors). The `googleDiscovery → "google"` | |
| 93 | − entries in React are compensating, not dangling. The closed SourceType | |
| 94 | − switch must map `"google"`/`"googleDiscovery"` → openapi or degraded rows | |
| 95 | − become hard failures. | |
| 96 | −3. **FumaDB is not versioned in practice.** All hosts pin `version: "1.0.0"`; | |
| 97 | − runtime bring-up is `ensureDrizzleRuntimeSchemaFromTables` (CREATE TABLE IF | |
| 98 | − NOT EXISTS + additive nullable ALTERs). New tables/nullable columns are free | |
| 99 | − on local/self-host/D1. **Cloud is the exception**: hand-edit | |
| 100 | − `apps/cloud/src/db/executor-schema.ts` + drizzle-kit migration + | |
| 101 | − `db:migrate:prod`; `db.schema.test.ts` enforces the mirror, so "forgot | |
| 102 | − cloud" fails tests. | |
| 103 | −4. **93 e2e files import `composePluginApi`** purely to type the client. Keep | |
| 104 | − it as a deprecated argument-ignoring shim returning the static | |
| 105 | − `FullExecutorApi`; sweep the call sites mechanically in one final PR. | |
| 106 | −5. **Keep the `plugin_id` column name** (four tables + blob namespace format | |
| 107 | − `o:<tenant>/<plugin_id>`). Renaming to `source_kind` is a cross-host | |
| 108 | − migration for zero behavior. Retype on the read side only. | |
| 109 | −6. **URL contracts:** `/integrations/add/$pluginKey` is navigated directly by | |
| 110 | − e2e; `/plugins/desktop-settings/` is asserted by the desktop-packaged suite | |
| 111 | − and baked into a tool's returned URL. Keep shapes / add redirects. | |
| 112 | −7. **CI gaps:** PRs run cloud (4 shards) + selfhost e2e only; local/desktop | |
| 113 | − suites gate push-to-main. PRs touching desktop-settings or the local loader | |
| 114 | − need a manual local e2e run pre-merge. | |
| 115 | −8. **Toolkit selection filters _connections_, not just tools** | |
| 116 | − (`connectionsList` goes through the active policy provider). Preserve when | |
| 117 | − moving toolkits to core, or the toolkit e2e suites go red. | |
| 118 | − | |
| 119 | −## Phase 0 — declutter (3 PRs) | |
| 120 | − | |
| 121 | −- **0.1** Delete empty `packages/plugins/{google,microsoft}`; drop the unused | |
| 122 | − `@executor-js/config` dep from openapi/mcp/graphql; move | |
| 123 | − `provider-service-split` to `packages/migrations/` (keep package name — it's | |
| 124 | − imported by three host data-migration registries). In the React maps, keep | |
| 125 | − `google`/`googleDiscovery` → openapi with a comment citing constraint 2. | |
| 126 | −- **0.2** Bake google/microsoft catalogs + spec-format adapters in as openapi | |
| 127 | − defaults; strip the re-threading from all four host configs. Gate: | |
| 128 | − `provider-plugins-ui`, `google-health-checks`, `microsoft-*` scenarios. | |
| 129 | −- **0.3** Move desktop-settings (438-line Electron-IPC client, not a | |
| 130 | − one-liner) to a first-class desktop-only route in `packages/app`; update the | |
| 131 | − static tool's URL and the desktop-packaged e2e selector; delete the plugin. | |
| 132 | − | |
| 133 | −## Phase 1 — data model (4 PRs) — FIRST PRIORITY | |
| 134 | − | |
| 135 | −Design choices (least-invasive, verified feasible): | |
| 136 | − | |
| 137 | −- The new **subject table is `tenant`-scoped, not owner-scoped** — readable by | |
| 138 | − any executor bound to the tenant with **zero policy changes** (the | |
| 139 | − `integration` table already proves the pattern). | |
| 140 | −- Cross-subject connection reads: add `reach: "bound" | "tenant"` to | |
| 141 | − `ExecutorOwnerPolicyContext` — widens `ownerVisibilityCondition` for reads | |
| 142 | − only; `assertOwnerWritable`/`assertOwnerPatch` untouched, so admin reach can | |
| 143 | − never write. Only two construction sites exist (`executor.ts`, | |
| 144 | − `test-config.ts`). | |
| 145 | −- Identity stays with the hosts (WorkOS / Better Auth member lists already | |
| 146 | − exist behind `AccountProvider.listMembers`); the subject table is the missing | |
| 147 | − **join** between host identity and executor connections, not a new identity | |
| 148 | − system. Tolerate `subject = null` (pure-org) and `subject = "local"`. | |
| 149 | − | |
| 150 | −PRs: | |
| 151 | − | |
| 152 | −- **1.1** Add tenant-scoped `subject` table (id, tenant, external_id, | |
| 153 | − created_at, last_seen_at, status) to `coreTables` + cloud mirror + one | |
| 154 | − drizzle migration. No readers yet. | |
| 155 | −- **1.2** Populate: upsert in `makeScopedExecutor` | |
| 156 | − (`packages/core/api/src/server/scoped-executor.ts` — every HTTP request and | |
| 157 | − MCP session on all four hosts passes through it) and at connection-create. | |
| 158 | − Add to `purgeOrganizationData` + its `TENANT_TABLES` test. | |
| 159 | −- **1.3** `reach` mode + `admin.listSubjectsWithConnections`-shaped read on a | |
| 160 | − derived handle. Policy test proving admin reach cannot write. Scope includes | |
| 161 | − (customer ask, owner.com 2026-07-27): per-subject reads ("connections for | |
| 162 | − subject X"), not just the aggregate list, and a privileged org-level API key | |
| 163 | − (keys already carry `{accountId, organizationId}` — | |
| 164 | − `apps/cloud/src/auth/api-keys.ts`) that resolves to the tenant-reach | |
| 165 | − read-only handle so customers can inspect connections programmatically. | |
| 166 | −- **1.4** Connection lifecycle: nullable `status` column | |
| 167 | − (active/disconnected/revoked/expired) plus `status_changed_at`; disconnect | |
| 168 | − writes status instead of deleting. Wide blast radius — status predicate | |
| 169 | − through `findConnectionRow`, `connectionsList`, credential resolution, | |
| 170 | − `toolsList`, org purge; own e2e sweep (`connections-*`, `health-checks*`, | |
| 171 | − `oauth-*`). PRIORITY RAISED (customer, 2026-07-28): the customer's | |
| 172 | − block-startup-until-reconnect flow needs connected-then-disconnected to be | |
| 173 | − distinguishable from never-connected, which hard-delete makes impossible | |
| 174 | − today; expose status + statusChangedAt through the admin API when this | |
| 175 | − lands. | |
| 176 | − | |
| 177 | −Toolkits data does NOT move in this phase (it moves with its reader in 2.4 — | |
| 178 | −moving it earlier means either rewriting the plugin storage layer twice or a | |
| 179 | −forbidden dual read path). | |
| 180 | − | |
| 181 | −### Framing: product view vs platform view (Rhys, 2026-07-27) | |
| 182 | − | |
| 183 | −The two-view model for consumer-facing surfaces, longer-term: | |
| 184 | − | |
| 185 | −- **Product view** — the bound, per-individual executor every request gets | |
| 186 | − today (their connections, their tools). The default; unchanged. | |
| 187 | −- **Platform view** — an operator-level view attached alongside it: read-only, | |
| 188 | − cross-user, explicitly reached for with elevated access (an escape hatch, | |
| 189 | − never a mode product view drifts into). The `reach: "bound" | "tenant"` | |
| 190 | − split in 1.3 is this model's first implementation; 1.5's admin API and the | |
| 191 | − privileged key are its first surface. Later platform-view candidates (usage, | |
| 192 | − policy overview, cross-user health) land on the same handle. | |
| 193 | − | |
| 194 | −Cloud wrinkle: cloud consumers like owner.com don't bring their own owner ids — | |
| 195 | −their users authenticate through our WorkOS, so `subject` is _our_ accountId | |
| 196 | −and the customer correlates on their side. The opaque-owner-ids-in-the- | |
| 197 | −customer's-id-space story fully applies only to embedded/API consumers; for | |
| 198 | −cloud, the platform view (+ `external_id` on the subject table) is what makes | |
| 199 | −the correlation tractable. | |
| 200 | − | |
| 201 | −### Fast path: owner.com admin-visibility ask (ships from Phase 1 alone) | |
| 202 | − | |
| 203 | −The owner.com ask (admin inspects users' connections; API key that can do the | |
| 204 | −same; available-vs-connected icon grid; direct connect links) needs only | |
| 205 | −1.1 → 1.2 → 1.3 plus a thin delivery layer — no dependency on Phase 0 or the | |
| 206 | −plugin kill. Delivery PRs on top of 1.3: | |
| 207 | − | |
| 208 | −- **1.5** Public admin API surface: `GET .../admin/users`, | |
| 209 | − `GET .../admin/users/:externalId/connections` (integration slug + status + | |
| 210 | − last_health only — never item_ids/oauth fields), gated on the privileged key | |
| 211 | − / admin membership. Public paths say `users`, not `subjects`/`tenants` — see | |
| 212 | − "Vocabulary" below. The available-vs-connected diff is computable client-side | |
| 213 | − from this plus the existing integration catalog. | |
| 214 | −- **1.6** Connect deep links: stable | |
| 215 | − `/connect/:integrationSlug`-shaped URL that lands the arriving user in the | |
| 216 | − connect flow for that integration (auth-aware redirect). Uses integration | |
| 217 | − slugs, not plugin routes, so it survives Phase 2.5 unchanged. | |
| 218 | − | |
| 219 | − 1.4 (lifecycle status) upgrades the grid from binary connected/not to | |
| 220 | − expired/disconnected states but is not a blocker for first ship. | |
| 221 | − | |
| 222 | −## Phase 2 — kill the registry (6 PRs) | |
| 223 | − | |
| 224 | −- **2.1** Remove dynamic loading: delete `load-plugins.ts` from | |
| 225 | − `@executor-js/config` (keep the rest — jsonc integration config is a separate | |
| 226 | − feature), drop `loadPluginsFromJsonc` from `apps/local`, delete | |
| 227 | − `packages/core/vite-plugin`, replace `virtual:executor/plugins-client` with a | |
| 228 | − static module in `packages/app` (5 `__root.tsx` importers + 3 ambient | |
| 229 | − `.d.ts`), edit `turbo.json` `dev.dependsOn` + root `prepare`, fix | |
| 230 | − `packages/app/vite.config.ts`'s dangling jsonc path. Breaking changeset. | |
| 231 | −- **2.2** Static route list: `composePluginApi` becomes the deprecated shim | |
| 232 | − (constraint 4); hosts stop passing plugin tuples; delete | |
| 233 | − `apps/{cloud,host-selfhost}/src/plugins.ts` (exist only to hold | |
| 234 | − `PluginExtensionServices` typing). | |
| 235 | −- **2.3** The switch: closed internal `SourceType` interface implemented by | |
| 236 | − openapi/graphql/mcp/**apps (current model, unchanged)** as plain libraries, | |
| 237 | − replacing `runtimes.get`. All four kinds convert together — converting three | |
| 238 | − and leaving apps on the plugin path means carrying dual dispatch; rewriting | |
| 239 | − the apps model in the same PR is what killed the rework. Map | |
| 240 | − `"google"`/`"googleDiscovery"` → openapi. `plugin_id` column name unchanged. | |
| 241 | −- **2.4** Toolkits to core: core tables + one-off migration out of | |
| 242 | − `plugin_storage` (enumerated invariants, no dual path), `toolPolicyProvider` | |
| 243 | − slot → native policy input (one call site, one implementation, one test), | |
| 244 | − ordinary core routes, `activeToolkitSlug` becomes an executor option. | |
| 245 | − Preserve the connection-visibility filter (constraint 8). | |
| 246 | −- **2.5** React: four named kind renderers; move plugin React code (~2.9k LoC, | |
| 247 | − e.g. `AddMcpIntegration.tsx`) into `packages/react` — watch the peer-dep | |
| 248 | − edge for cycles; delete `defineClientPlugin`, the plugins context, both | |
| 249 | − `KIND_TO_PLUGIN_KEY` maps, and `/plugins/$pluginId/$` (with redirects for | |
| 250 | − desktop-settings + toolkits). Keep `/integrations/add/$pluginKey` URL shape. | |
| 251 | − Don't forget the second axis: `useSecretProviderPlugins` (secrets page). | |
| 252 | −- **2.6** Detach `CredentialProvider` from `PluginSpec` (the interface in | |
| 253 | − `provider.ts` is already narrow — secret-store plugins keep it as their | |
| 254 | − seam); delete `plugin.ts`, `plugin-routes.ts`, `plugin-example`, and the | |
| 255 | − shim; mechanical sweep of the 93 e2e files; changesets for all seven | |
| 256 | − published packages. | |
| 257 | − | |
| 258 | −## Phase 3 — custom = concept package (3–4 PRs) — NOW BEFORE PHASE 2 | |
| 259 | − | |
| 260 | −(See RESEQUENCED note at top: lands side by side with the protocol plugins, | |
| 261 | −inside the apps plugin's existing costume; Phase 2 kills the registry | |
| 262 | −afterward, with apps converting in 2.3 as an already-stable model.) | |
| 263 | − | |
| 264 | −Port the rework's app model in place as the `apps` source kind's new model: | |
| 265 | −published source, manifest derived by execution, content-hashed immutable | |
| 266 | −versions, rollback = pointer move. New tables are additive greenfield (same | |
| 267 | −pattern as 1.1: coreTables + cloud mirror + migration, no readers first). | |
| 268 | −Existing app sources migrate one-off to published-source form — no dual model | |
| 269 | −within the apps kind. Then the OpenAPI/MCP generators + `GeneratorClaim` as an | |
| 270 | −optional on-ramp (spec → generated app source); GraphQL generator later. | |
| 271 | −Long-term option (explicitly out of scope): protocol kinds themselves become | |
| 272 | −generators and "custom" is the only kind. | |
| 273 | − | |
| 274 | −### Feasibility findings (two deep read-throughs, 2026-07-28) | |
| 275 | − | |
| 276 | −**Current plugin** (~7.6k LoC): only ~2.65k encodes the old model (store | |
| 277 | −collections, sync state machine, publish/discover/descriptor, routes, | |
| 278 | −console client). ~2.6k is model-agnostic and survives verbatim: all three | |
| 279 | −sandbox executors, the bundler stack, bindings/bridge, authoring + | |
| 280 | −standard-schema, conformance. ~1.9k (hand-rolled git client + source | |
| 281 | −fetchers) hangs on the publish-input decision below. Storage is entirely | |
| 282 | −generic `plugin_storage` rows + content-addressed blobs — zero apps-specific | |
| 283 | −tables or migrations exist, so new tables are pure greenfield. Cloud prod is | |
| 284 | −git-source-only; **source text is never persisted today (only compiled | |
| 285 | −bundles)**, so the one-off migration re-fetches source from the recorded git | |
| 286 | −URLs (fetcher exists). Load-bearing invariants to preserve: tool addresses | |
| 287 | −`tools.<app>.org.published.<name>` + the `<fileSlug>__<key>` naming rule, | |
| 288 | −app-slug↔integration-slug 1:1 guard, publish CAS | |
| 289 | −(`AppPublishConflictError`), tombstone-not-delete, token-in-provider-item | |
| 290 | −(never in the source row). Safety net: `custom-tools.test.ts` (534-line | |
| 291 | −cloud+selfhost console loop), local directory-picker, packed-CLI, desktop | |
| 292 | −e2e, ~2.9k LoC in-package tests. Pre-existing gaps flagged: R2 blobs never | |
| 293 | −GC'd and org purge misses them; cloud silently falls back to in-process | |
| 294 | −executor without the worker loader; host-cloudflare doesn't register apps. | |
| 295 | − | |
| 296 | −**Rework model** (portable tiers): T1 model proper ~3.9k LoC (authoring SDK, | |
| 297 | −publish-by-execution, projection, generator verification, storage port, | |
| 298 | −content hashing); T2 generators ~3.2k (fromOpenAPI 1k — complete; fromMCP | |
| 299 | −0.6k — computed group with real streamable-HTTP client; json-schema engine; | |
| 300 | −byte-for-byte GeneratorClaim verification; render endpoint); T4 host/registry | |
| 301 | −~2.9k explicitly skipped. Authoring is Standard Schema | |
| 302 | −(StandardSchemaV1 & StandardJSONSchemaV1) with raw-JSON-Schema literals as a | |
| 303 | −typed first-class alternative — matches our rule. Effect beta.93→beta.59 | |
| 304 | −delta is small (ConstraintDecoder → one-line fix; hand-roll | |
| 305 | −StandardJSONSchemaV1 ~10 lines; reuse core sha256Hex). NOT portable as-is: | |
| 306 | −rework `storage-db.ts` (write a new adapter over our tables against the | |
| 307 | −~90-line Storage port interface) and `engine.ts` (extract only ~700 LoC of | |
| 308 | −model-owned logic: computed-surface cache + requirement resolution + | |
| 309 | −runTool). Known rework gaps to fix at port time, not inherit: derive-time | |
| 310 | −execution is unsandboxed in-process (`code-executor.ts:23` records it) — our | |
| 311 | −port MUST run publish-time evaluation in the existing sandbox `collect` path | |
| 312 | −from day one; fan-in computed groups unsupported; OAuth discovery | |
| 313 | −unsupported. | |
| 314 | − | |
| 315 | −**Semantic deltas — decided by Rhys before 3.1 specs:** | |
| 316 | − | |
| 317 | −1. **Handler credential contract (the deep one).** Current: handlers get an | |
| 318 | − opaque callable `AppIntegrationClient` proxy that re-enters | |
| 319 | − `ctx.execute` — app code never sees credentials, can only call other | |
| 320 | − executor tools. Rework (D49): engine resolves credentials once and hands | |
| 321 | − the handler **credential fields as data** (`ResolvedConnection.fields`, | |
| 322 | − e.g. access_token) for direct HTTP — that's what makes generators | |
| 323 | − possible (generated code does raw fetch with auth) and what | |
| 324 | − `defineProvider` exists for. Not either/or in principle (a tool-call | |
| 325 | − bridge can coexist as a second requirement kind later), but the ported | |
| 326 | − model's primary contract is fields-as-data per the rework. | |
| 327 | −2. **Publish input.** Rework publish takes `SourceFiles` (push-based). | |
| 328 | − Decision: keep git/local-directory as _fetchers that produce | |
| 329 | − SourceFiles_ (preserves prod sources, the e2e suite, and the migration | |
| 330 | − path; git-client survives), with direct push publish added as the new | |
| 331 | − API. Delete-git-later stays open. | |
| 332 | −3. **App identity** moves from caller-supplied slug to the `defineApp` | |
| 333 | − export inside the source (rework shape). | |
| 334 | −4. **Versioning**: content hash of source IS the version identity; the | |
| 335 | − caller-supplied `sourceRef` + descriptor v6 die with the old model. | |
| 336 | −5. **Ownership**: app/version rows carry `space` and NO owner — spaces | |
| 337 | − curate apps, principals own custody (connections stay on the legacy | |
| 338 | − owner model in this phase; the opaque-OwnerRef swap is Phase 4). | |
| 339 | − | |
| 340 | −### Phase 3 PR sequence (revised) | |
| 341 | − | |
| 342 | −- **3.1** Port the model core as internal packages, no readers: authoring | |
| 343 | − SDK + runtime brands, publish pipeline (evaluation routed through the | |
| 344 | − sandbox `collect` seam), Storage port + adapter, greenfield tables | |
| 345 | − (`app`, `app_version`, `computed_surface` — space column, content-hash | |
| 346 | − identity, `current` pointer) in coreTables + cloud mirror + migration. | |
| 347 | − Ported rework tests come along (authoring-boundary, tool-groups, | |
| 348 | − publish). | |
| 349 | −- **3.2** Swap the apps plugin's internals onto the new model behind the | |
| 350 | − same costume: resolveTools/projectToolSchema/invokeTool read the new | |
| 351 | − tables; publish endpoint (`POST` source files); setCurrent/rollback + | |
| 352 | − versions listing routes; git/local-dir sync becomes fetch→publish. | |
| 353 | − Tool addresses and slug guards preserved; old collections | |
| 354 | − (`apps_descriptors`/`apps_tools`/`apps_sources`) still written by | |
| 355 | − NOTHING — one-off migration converts existing sources (re-fetch from | |
| 356 | − git), then the collections are dead. e2e: full existing custom-tools | |
| 357 | − suite green + new version-history/rollback scenario. | |
| 358 | −- **3.3** Console UI for the new model (sources → apps with versions, | |
| 359 | − publish status, rollback) replacing the ~1.3k LoC old client/panels. | |
| 360 | −- **3.4** Generators: fromOpenAPI + fromMCP + GeneratorClaim verification + | |
| 361 | − render endpoint, surfaced as the "add from spec/server" on-ramp. | |
| 362 | − (Vendors google/microsoft conversion ~1.15k LoC ports later with 0.2.) | |
| 363 | − | |
| 364 | −## Spaces: the rework's tenancy primitive (added Rhys, 2026-07-28) | |
| 365 | − | |
| 366 | −The rework's SDK has exactly one tenancy concept: `SpaceId` | |
| 367 | −(`.reference/executor-rework/docs/TENANCY.md`, `docs/MODEL.md`). Adopt it as | |
| 368 | −the end-state tenancy model here. The shape: | |
| 369 | − | |
| 370 | −- **Flat, no hierarchy, no default.** No parent, no inheritance, no | |
| 371 | − distinguished rows (rework D39/D44b). Sharing is an explicit act, never a | |
| 372 | − tree walk (R2 — re-ruled twice, do not relitigate). Every scoped row | |
| 373 | − carries a `space` column; every space-scoped route carries the space in | |
| 374 | − the URL path, never ambient context. | |
| 375 | −- **No org/project/team/user below the deployment edge.** The engine answers | |
| 376 | − "what is in this space"; the deployment answers "who is this and which | |
| 377 | − spaces may they touch" through named seams (`authorizeBrowserResource`, | |
| 378 | − `dashboardSpacesForPrincipal`, MCP space resolver). Membership is | |
| 379 | − authorized LIVE per request at the edge — projection may be cached, | |
| 380 | − authorization never. | |
| 381 | −- **Deployment mappings:** cloud = one WorkOS org mirrors lazily to one | |
| 382 | − space (mirror row `workosOrgId → space` is also the billing join — Autumn | |
| 383 | − keys on workosOrgId, never a space); self-host = one instance, one | |
| 384 | − found-or-created "Default" space, Better Auth org repurposed as the | |
| 385 | − instance membership table; local/desktop = one "local" space, one static | |
| 386 | − principal, single-tenancy declared via explicit sentinel, not defaulted. | |
| 387 | −- **Spaces curate, principals own.** Custody (secrets, connections' | |
| 388 | − secret-bearing halves) belongs to principals via opaque `OwnerRef` + | |
| 389 | − per-request `Reach` (R3); spaces curate apps and hold rows. Publishing an | |
| 390 | − app into a space IS its presence there — no installation rows, no | |
| 391 | − bindings; cross-space arrival (sharing/registry) is a future share-edge | |
| 392 | − row, not golden-path ceremony (D45/D42). | |
| 393 | −- Cross-org OAuth callbacks complete into the _initiating_ org's space; the | |
| 394 | − org header is a routing hint, not an authorization statement (R7). | |
| 395 | − | |
| 396 | −**DECIDED (Rhys, 2026-07-28): spaces land in the CORE PRODUCT NOW — Phase S, | |
| 397 | −before the apps port.** Apps then publishes into real spaces from birth. | |
| 398 | − | |
| 399 | −**Mapping onto this repo — the no-migration trick:** today's `tenant` column | |
| 400 | −is de facto the space — cloud writes the WorkOS org id there, self-host/local | |
| 401 | −write one constant. So: keep the persisted `tenant` column name (same rule as | |
| 402 | −`plugin_id`), and declare that **its value IS the space id**. Each org's | |
| 403 | −first space is grandfathered with `spaceId = <historical tenant value>` (the | |
| 404 | −org id); NEW spaces mint fresh ids written into the same column. The org→ | |
| 405 | −space mirror table records which space ids belong to which org (and carries | |
| 406 | −the billing join — Autumn keys on workosOrgId, never a space). Existing rows | |
| 407 | −never move; multi-space needs no schema change on scoped tables. "Space" is | |
| 408 | −contract vocabulary; "tenant" retreats to cloud-internal (see Vocabulary). | |
| 409 | − | |
| 410 | −### Phase S — spaces in core (before apps) | |
| 411 | − | |
| 412 | −- **S1 — space table + boot ensure (no readers).** Greenfield `space` table | |
| 413 | − (id, name, created_at) in coreTables + cloud mirror + migration, plus the | |
| 414 | − cloud `org→space` mirror table. Boot/first-contact ensures the degenerate | |
| 415 | − space per deployment: cloud lazily on first authenticated org contact | |
| 416 | − (grandfathered id = org id), self-host "Default", local "local". Same | |
| 417 | − no-readers pattern as PR 1.1. | |
| 418 | −- **S2 — space threading.** `SpaceId` brand in the SDK; the scoped-executor | |
| 419 | − seam resolves request → space id (today: identical to tenant) and stamps | |
| 420 | − it; reads interpret `tenant` columns as space ids. No behavior change — | |
| 421 | − this is the retype + vocabulary PR. `reach: "tenant"` reads as | |
| 422 | − "space-wide reach" from here on. | |
| 423 | −- **S3 — multi-space on cloud + product surface.** Create/rename space, | |
| 424 | − space switcher in the console, membership = org membership (all spaces of | |
| 425 | − the org for now), org→N mirror rows, space-scoped catalog/connection | |
| 426 | − views (rows follow automatically — they're keyed by the space id in the | |
| 427 | − tenant column). Billing stays org-keyed via the mirror. e2e: two spaces | |
| 428 | − in one org, isolation of catalogs/connections between them, switcher. | |
| 429 | −- **S4 — sharing model on spaces.** Connect-time "share with space" writes | |
| 430 | − the space's shared owner-ref (replaces the org tier semantically; the | |
| 431 | − persisted `owner: "org"|"user"` column keeps its name, read-side | |
| 432 | − reinterpretation only). Edge computes reach as | |
| 433 | − [principal, sharedRef(space) for each member space]. This is the bridge | |
| 434 | − to Phase 4's full owner-set reach. | |
| 435 | −- **Later (not scheduled): local multi-space, file-backed.** Each local | |
| 436 | − space is its own store file (portable — copy/backup/sync/check in | |
| 437 | − alongside a repo); the daemon opens one storage handle per space and | |
| 438 | − routes by space id; the engine still sees flat SpaceIds. Custody never | |
| 439 | − enters the space file — secrets stay in the principal-keyed provider | |
| 440 | − (keychain), so a copied file carries connection metadata, never tokens. | |
| 441 | −- Phase 4 completes the picture: space-in-path routes, | |
| 442 | − `dashboardSpacesForPrincipal`-style seams, reach as an explicit owner | |
| 443 | − set. | |
| 444 | − | |
| 445 | −Apps (Phase 3) then lands after S1/S2 — app rows carry a real `space` FK | |
| 446 | −from birth; S3/S4 can proceed in parallel with the apps port. | |
| 447 | − | |
| 448 | −Interaction with shipped Phase 1 work: the subject table and platform view | |
| 449 | −are edge/deployment concepts (who exists in this identity world, what may | |
| 450 | −an operator read) — they sit above spaces and survive unchanged; `reach: | |
| 451 | −"tenant"` becomes "space-wide reach" vocabulary-wise at Phase 4. | |
| 452 | − | |
| 453 | −## Vocabulary: retire "tenant" from consumer-facing surfaces (Rhys, 2026-07-27) | |
| 454 | − | |
| 455 | −Direction: the SDK should be agnostic to the consumer's data-model shape, the | |
| 456 | −way Composio's is — consumers pass an opaque user/owner id string and hang | |
| 457 | −whatever hierarchy they have (user, team, workspace) off it; the SDK never | |
| 458 | −models their org chart. The rework already designed this (`OwnerRef` opaque | |
| 459 | −brand, `Reach = {owners}`, deployments translate identity at the edge) — adopt | |
| 460 | −its vocabulary, not its schedule. "Tenant" remains a real _internal_ concept | |
| 461 | −(which Executor-cloud org owns the rows) but must not leak into surfaces | |
| 462 | −consumers touch. | |
| 463 | − | |
| 464 | −Rules, effective immediately: | |
| 465 | − | |
| 466 | −- Persisted column names (`tenant`, `subject`, `owner`) stay — renaming | |
| 467 | − storage is a cross-host migration for zero behavior (same rule as | |
| 468 | − `plugin_id`). | |
| 469 | −- Internal policy code (`reach: "tenant"`, `ExecutorOwnerPolicyContext`) may | |
| 470 | − keep the words. | |
| 471 | −- One contract noun: **`owner`** — the opaque ref naming who a connection | |
| 472 | − belongs to (end-user, team, workspace, or the org itself; the rework's | |
| 473 | − `OwnerRef`). "User" cannot be the contract noun because org-shared | |
| 474 | − connections are a load-bearing feature an end-user word can't cover. | |
| 475 | −- **"user" is presentation vocabulary only**, for surfaces whose rows genuinely | |
| 476 | − are the customer's end-users — the Phase 1.5 admin API says `users` because | |
| 477 | − the subject table is populated from authenticated principals. | |
| 478 | −- Hazard: the persisted `owner` column keeps its old meaning (the | |
| 479 | − `"org" | "user"` tier discriminator), while Phase 4 introduces the opaque | |
| 480 | − ref as a distinctly named type (`OwnerRef`). Same word, two meanings, | |
| 481 | − separated by layer — say so in policy-layer comments. | |
| 482 | −- The full swap (opaque owner refs at the SDK boundary, tenant retreating to a | |
| 483 | − cloud-internal detail) lands with the Phase 4 minimal SDK. | |
| 484 | − | |
| 485 | −## Phase 4 — minimal SDK + platform/product split | |
| 486 | − | |
| 487 | −New consumer boundary modeled on the rework: `createExecutor` / | |
| 488 | −`createRemoteExecutor`, no Effect at the boundary, `tools.list/execute` + | |
| 489 | −connections + the new users/admin surface; policies/toolkits become optional | |
| 490 | −layers consumers opt into, not baked opinions (Composio-style minimalism). | |
| 491 | − | |
| 492 | −Authorization end-state (Rhys, 2026-07-27): the engine enforces scope | |
| 493 | −mechanically but never decides policy — the host computes, per request, the | |
| 494 | −set of owners the request may read (the rework's `Reach = { owners }`), and | |
| 495 | −the storage predicate is just `row.owner ∈ reach.owners`. The 1.3 | |
| 496 | −`reach: "bound" | "tenant"` enum is a stepping stone and is replaced by this | |
| 497 | −set shape here: "bound" = a reach of one, "tenant" = all owners (wildcard so | |
| 498 | −cloud needn't enumerate), and in-between cases (team views, support sessions, | |
| 499 | −subset-restricted org keys) are just different sets. Writes stay hard-pinned | |
| 500 | −to a single owner; only read reach generalizes. Trust model becomes a | |
| 501 | −per-deployment choice: a trusted backend with a server key gets tenant-wide | |
| 502 | −reach and layers its own authorization (Composio-style); an end-user session | |
| 503 | −gets a reach of one. The enum-to-set swap is contained: the policy context is | |
| 504 | −constructed in exactly two places. | |
| 505 | −Start the platform/product package separation _during_ 2.3–2.6 (each PR moves | |
| 506 | −one thing to its final home) rather than as a big bang; the SDK boundary | |
| 507 | −itself lands last. | |
| 508 | − | |
| 509 | −## Bookkeeping | |
| 510 | − | |
| 511 | −- Create `MISTAKES.md` / `DESIRES.md` / `LEARNINGS.md` (required by AGENTS.md, | |
| 512 | − currently missing) with the first entries from this migration. | |
| 513 | −- `apps/local/src/db/executor-schema.ts` is already stale vs `coreTables` | |
| 514 | − (missing several columns); its "keep in sync" comment is untrustworthy — | |
| 515 | − runtime schema comes from the ensure path, not this file. | |