History

Adopt JSON-first document schema in SQLite persistence plan

- Replace normalized table proposal with a `documents` envelope + `data_json` model - Update indexing, migration phases, test scope, and done criteria for document-based persistence - Add schema-versioning and JSON shape drift mitigations

Julius Marminge committed 8 months ago

Commit bf9f6531 parent e7e289f1 file · +50 −108Browse files

Unified · Split

.plans/11-centralized-sqlite-persistence-revamp.md+50−108
@@ -88,101 +88,24 @@ Provider remains the runtime execution engine. PersistenceService becomes the du · 87 unmodified lines
8888 ## Core Domain Model (SQLite)
8989 Use strict schema + migrations (e.g. `PRAGMA user_version` or migration table), WAL mode, and transactional writes.
9090
91−### Proposed tables
92−1. `projects`
93− - `id TEXT PRIMARY KEY`
94− - `cwd TEXT NOT NULL UNIQUE` (normalized)
95− - `name TEXT NOT NULL`
96− - `model TEXT NOT NULL`
97− - `expanded INTEGER NOT NULL DEFAULT 1`
98− - `created_at TEXT NOT NULL`
99− - `updated_at TEXT NOT NULL`
91+### JSON-first (iteration-friendly) schema
92+Use a document-style core table with a minimal indexed envelope. Put evolving entity fields in `data_json`.
10093
101−2. `project_scripts`
102− - `project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE`
103− - `script_id TEXT NOT NULL`
104− - `name TEXT NOT NULL`
105− - `command TEXT NOT NULL`
106− - `icon TEXT NOT NULL`
107− - `run_on_worktree_create INTEGER NOT NULL`
108− - `position INTEGER NOT NULL`
109− - `PRIMARY KEY (project_id, script_id)`
110−
111−3. `threads`
112− - `id TEXT PRIMARY KEY` (client thread id)
113− - `project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE`
114− - `codex_thread_id TEXT NULL` (runtime provider thread id)
115− - `title TEXT NOT NULL`
116− - `model TEXT NOT NULL`
94+1. `documents`
95+ - `id TEXT PRIMARY KEY`
96+ - examples: `project:<uuid>`, `thread:<uuid>`, `message:<uuid>`, `turn_summary:<threadId>:<turnId>`
97+ - `kind TEXT NOT NULL`
98+ - `project` | `thread` | `message` | `turn_summary`
99+ - `project_id TEXT NULL`
100+ - `thread_id TEXT NULL`
101+ - `sort_key INTEGER NULL`
102+ - message order, thread ordering hints, summary ordering, etc.
103+ - `schema_version INTEGER NOT NULL DEFAULT 1`
117104 - `created_at TEXT NOT NULL`
118− - `last_visited_at TEXT NULL`
119− - `branch TEXT NULL`
120− - `worktree_path TEXT NULL`
121− - `terminal_open INTEGER NOT NULL DEFAULT 0`
122− - `terminal_height INTEGER NOT NULL DEFAULT 280`
123− - `active_terminal_id TEXT NOT NULL DEFAULT 'default'`
124− - `active_terminal_group_id TEXT NOT NULL DEFAULT 'group-default'`
125105 - `updated_at TEXT NOT NULL`
106+ - `data_json TEXT NOT NULL`
126107
127−4. `thread_terminals`
128− - `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
129− - `terminal_id TEXT NOT NULL`
130− - `position INTEGER NOT NULL`
131− - `PRIMARY KEY (thread_id, terminal_id)`
132−
133−5. `thread_terminal_groups`
134− - `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
135− - `group_id TEXT NOT NULL`
136− - `position INTEGER NOT NULL`
137− - `PRIMARY KEY (thread_id, group_id)`
138−
139−6. `thread_terminal_group_members`
140− - `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
141− - `group_id TEXT NOT NULL`
142− - `terminal_id TEXT NOT NULL`
143− - `position INTEGER NOT NULL`
144− - `PRIMARY KEY (thread_id, group_id, terminal_id)`
145−
146−7. `messages`
147− - `id TEXT PRIMARY KEY`
148− - `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
149− - `role TEXT NOT NULL` (`user`/`assistant`)
150− - `text TEXT NOT NULL`
151− - `created_at TEXT NOT NULL`
152− - `streaming INTEGER NOT NULL DEFAULT 0`
153− - `provider_item_id TEXT NULL`
154− - `turn_id TEXT NULL`
155− - `position INTEGER NOT NULL`
156−
157−8. `message_attachments`
158− - `message_id TEXT NOT NULL REFERENCES messages(id) ON DELETE CASCADE`
159− - `attachment_id TEXT NOT NULL`
160− - `type TEXT NOT NULL`
161− - `name TEXT NOT NULL`
162− - `mime_type TEXT NOT NULL`
163− - `size_bytes INTEGER NOT NULL`
164− - `position INTEGER NOT NULL`
165− - `PRIMARY KEY (message_id, attachment_id)`
166−
167−9. `thread_turn_summaries`
168− - `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
169− - `turn_id TEXT NOT NULL`
170− - `completed_at TEXT NOT NULL`
171− - `status TEXT NULL`
172− - `assistant_message_id TEXT NULL`
173− - `checkpoint_turn_count INTEGER NULL`
174− - `PRIMARY KEY (thread_id, turn_id)`
175−
176−10. `thread_turn_diff_files`
177− - `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
178− - `turn_id TEXT NOT NULL`
179− - `path TEXT NOT NULL`
180− - `kind TEXT NULL`
181− - `additions INTEGER NULL`
182− - `deletions INTEGER NULL`
183− - `PRIMARY KEY (thread_id, turn_id, path)`
184−
185−11. `provider_events`
108+2. `provider_events`
186109 - `seq INTEGER PRIMARY KEY AUTOINCREMENT`
187110 - `id TEXT NOT NULL UNIQUE`
188111 - `session_id TEXT NOT NULL`
@@ -199,27 +122,43 @@ Use strict schema + migrations (e.g. `PRAGMA user_version` or migration table), · 10 unmodified lines
199122 - `payload_json TEXT NULL`
200123 - `created_at TEXT NOT NULL`
201124
202−12. `state_events`
125+3. `state_events`
203126 - `seq INTEGER PRIMARY KEY AUTOINCREMENT`
204127 - `event_type TEXT NOT NULL` (e.g. `project.upsert`, `thread.delete`, `message.upsert`)
205128 - `entity_id TEXT NOT NULL`
206129 - `payload_json TEXT NOT NULL`
207130 - `created_at TEXT NOT NULL`
208131
209−13. `metadata`
132+4. `metadata`
210133 - `key TEXT PRIMARY KEY`
211134 - `value_json TEXT NOT NULL`
212− - for one-time migrations and import markers
135+ - for one-time migrations/import markers
213136
214−### Key indexes
215−- `projects(cwd)`
216−- `threads(project_id, created_at DESC)`
217−- `messages(thread_id, position)`
137+### Envelope indexes
138+- `documents(kind)`
139+- `documents(project_id, kind)`
140+- `documents(thread_id, kind, sort_key)`
141+- `documents(kind, updated_at DESC)`
218142 - `provider_events(session_id, seq)`
219143 - `provider_events(thread_id, seq)`
220−- `thread_turn_summaries(thread_id, completed_at DESC)`
221144 - `state_events(seq)`
222145
146+### Document conventions
147+1. Validate each `kind` with Zod before writes and after reads.
148+2. Keep envelope fields stable (`kind`, `project_id`, `thread_id`, `sort_key`) for fast list/filter queries.
149+3. Put fast-changing/evolving structure in `data_json`:
150+ - project: name/cwd/model/expanded/scripts
151+ - thread: title/model/codex thread id/terminal layout/branch/worktree
152+ - message: role/text/attachments/streaming/provider ids
153+ - turn_summary: completed/status/checkpoint turn count/file +/- summary
154+4. Use `schema_version` per document to support incremental document migrations without global table churn.
155+
156+### Normalization runway (later, optional)
157+When model shapes stabilize, we can materialize normalized tables from `documents` without changing the external protocol:
158+1. Add typed read models (`projects_v2`, `messages_v2`, etc.).
159+2. Backfill from `documents`.
160+3. Switch read paths gradually.
161+
223162 ## Streaming Protocol and Synchronization
224163 Keep current WS request/response envelope and push model. Extend with state snapshot + patch stream.
225164
@@ -416,14 +355,14 @@ Optional future mode (not required for this revamp): · 190 unmodified lines
416355 2. Add migration boot in `apps/server/src/index.ts`.
417356 3. Update default state path to `~/.t3/state.sqlite` (with env override).
418357
419−### Phase 1: Projects + threads in DB
420−1. Replace `ProjectRegistry` JSON backend with DB-backed repository.
358+### Phase 1: Document store cut-in (projects + threads first)
359+1. Replace `ProjectRegistry` JSON backend with DB-backed document repository.
421360 2. Add `threads.*` methods to server routes.
422361 3. Keep web reducer but stop writing to localStorage for these entities.
423362
424−### Phase 2: Message + turn persistence
363+### Phase 2: Message + turn document persistence
425364 1. Ingest provider events into DB (`provider_events`).
426−2. Project events into `messages` and `thread_turn_*`.
365+2. Project events into `documents` rows for `message` and `turn_summary` kinds.
427366 3. Add `state.event` push and `state.catchUp`.
428367
429368 ### Phase 3: Web client cutover
@@ -439,6 +378,7 @@ Optional future mode (not required for this revamp): · 9 unmodified lines
439378 ### Phase 5: Cleanup + compatibility
440379 1. Remove deprecated localStorage/persistence schema paths once migration is stable.
441380 2. Keep temporary migration fallback for one release window.
381+3. (Optional) add a follow-up normalization milestone when schemas settle.
442382
443383 ## Streaming/Projection Design Notes
444384 1. Durable append:
@@ -465,7 +405,7 @@ Backend changes require new tests and runtime validation. · 20 unmodified lines
465405
466406 ### Unit tests
467407 1. DB schema + migrations
468−2. Repository CRUD (projects/threads/messages)
408+2. Document repository CRUD (`project`/`thread`/`message`/`turn_summary`)
469409 3. Provider event projection logic
470410 4. Checkpoint summary parser (`@pierre/diffs` integration and malformed diff fallback)
471411 5. Keybindings JSON behavior remains unchanged (read/write + malformed-file handling)
@@ -498,10 +438,12 @@ Backend changes require new tests and runtime validation. · 26 unmodified lines
498438 - Mitigation: strict schema validation + partial-row rejection + migration markers + backups.
499439 5. Risk: checkpoint summary parsing overhead.
500440 - Mitigation: parse only finalized checkpoint ranges and fallback to lightweight stat extraction.
441+6. Risk: JSON shape drift across fast iterations.
442+ - Mitigation: per-kind Zod schemas + `schema_version` in documents + migration tests for older payloads.
501443
502444 ## Done Criteria
503445 1. No core durable app data depends on renderer localStorage.
504−2. Projects/threads/messages persist in `~/.t3/state.sqlite`.
446+2. Projects/threads/messages/turn summaries persist in `~/.t3/state.sqlite` via JSON-first documents.
505447 3. Checkpoint patch content remains Git-backed; optional summaries are in DB.
506448 4. Keybindings remain JSON-backed at `~/.t3/keybindings.json`.
507449 5. Theme/editor/last-invoked-script remain browser-scoped (or explicitly migrated in a separate opt-in feature).
@@ -513,8 +455,8 @@ Backend changes require new tests and runtime validation. · 5 unmodified lines
513455 - `apps/server/src/index.ts` (state db path + bootstrap)
514456 - `apps/server/src/wsServer.ts` (new state/thread routes + push channel)
515457 - `apps/server/src/providerManager.ts` (persist event hooks + checkpoint summary writes)
516−- `apps/server/src/projectRegistry.ts` (replace or wrap with DB repo)
517−- new persistence modules (`stateDb`, migrations, repositories)
458+- `apps/server/src/projectRegistry.ts` (replace or wrap with document-backed repo)
459+- new persistence modules (`stateDb`, migrations, document repository, persistence service)
518460
519461 ### Web
520462 - `apps/web/src/store.ts` (remove localStorage core persistence)
@@ -530,7 +472,7 @@ Backend changes require new tests and runtime validation. · 9 unmodified lines
530472
531473 ## Open Questions
532474 1. Should runtime mode (`full-access` vs `approval-required`) remain browser-scoped UX state, or be elevated to machine policy?
533−2. Should terminal panel layout/state be in SQLite (cross-device consistency) or preference JSON (human-editable)?
475+2. Should terminal panel layout/state be in SQLite (shared) or stay browser-scoped UX state?
534476 3. Do we need provider event retention limits from day one?
535477 4. Do we keep `.logs/threads/*.events.ndjson` as debug mirror after DB event log is live?
536478 5. For non-desktop/browser mode, should state still centralize to `~/.t3/state.sqlite` or support workspace-local override by default?