Adopt JSON-first document schema in SQLite persistence plan

- Replace normalized table proposal with a `documents` envelope + `data_json` model - Update indexing, migration phases, test scope, and done criteria for document-based persistence - Add schema-versioning and JSON shape drift mitigations

Julius Marminge committed 8 months ago

Commit bf9f6531 parent e7e289f1 file · +50 −108Browse files

Unified · Split

.plans/11-centralized-sqlite-persistence-revamp.md+50−108

@@ -88,101 +88,24 @@ Provider remains the runtime execution engine. PersistenceService becomes the du · 87 unmodified lines
88	## Core Domain Model (SQLite)	88	## Core Domain Model (SQLite)
89	Use strict schema + migrations (e.g. `PRAGMA user_version` or migration table), WAL mode, and transactional writes.	89	Use strict schema + migrations (e.g. `PRAGMA user_version` or migration table), WAL mode, and transactional writes.
90		90
91	### Proposed tables	91	### JSON-first (iteration-friendly) schema
92	1. `projects`	92	Use a document-style core table with a minimal indexed envelope. Put evolving entity fields in `data_json`.
93	- `id TEXT PRIMARY KEY`
94	- `cwd TEXT NOT NULL UNIQUE` (normalized)
95	- `name TEXT NOT NULL`
96	- `model TEXT NOT NULL`
97	- `expanded INTEGER NOT NULL DEFAULT 1`
98	- `created_at TEXT NOT NULL`
99	- `updated_at TEXT NOT NULL`
100		93
101	2. `project_scripts`	94	1. `documents`
102	- `project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE`	95	- `id TEXT PRIMARY KEY`
103	- `script_id TEXT NOT NULL`	96	- examples: `project:<uuid>`, `thread:<uuid>`, `message:<uuid>`, `turn_summary:<threadId>:<turnId>`
104	- `name TEXT NOT NULL`	97	- `kind TEXT NOT NULL`
105	- `command TEXT NOT NULL`	98	- `project` \| `thread` \| `message` \| `turn_summary`
106	- `icon TEXT NOT NULL`	99	- `project_id TEXT NULL`
107	- `run_on_worktree_create INTEGER NOT NULL`	100	- `thread_id TEXT NULL`
108	- `position INTEGER NOT NULL`	101	- `sort_key INTEGER NULL`
109	- `PRIMARY KEY (project_id, script_id)`	102	- message order, thread ordering hints, summary ordering, etc.
110		103	- `schema_version INTEGER NOT NULL DEFAULT 1`
111	3. `threads`
112	- `id TEXT PRIMARY KEY` (client thread id)
113	- `project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE`
114	- `codex_thread_id TEXT NULL` (runtime provider thread id)
115	- `title TEXT NOT NULL`
116	- `model TEXT NOT NULL`
117	- `created_at TEXT NOT NULL`	104	- `created_at TEXT NOT NULL`
118	- `last_visited_at TEXT NULL`
119	- `branch TEXT NULL`
120	- `worktree_path TEXT NULL`
121	- `terminal_open INTEGER NOT NULL DEFAULT 0`
122	- `terminal_height INTEGER NOT NULL DEFAULT 280`
123	- `active_terminal_id TEXT NOT NULL DEFAULT 'default'`
124	- `active_terminal_group_id TEXT NOT NULL DEFAULT 'group-default'`
125	- `updated_at TEXT NOT NULL`	105	- `updated_at TEXT NOT NULL`
		106	- `data_json TEXT NOT NULL`
126		107
127	4. `thread_terminals`	108	2. `provider_events`
128	- `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
129	- `terminal_id TEXT NOT NULL`
130	- `position INTEGER NOT NULL`
131	- `PRIMARY KEY (thread_id, terminal_id)`
132
133	5. `thread_terminal_groups`
134	- `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
135	- `group_id TEXT NOT NULL`
136	- `position INTEGER NOT NULL`
137	- `PRIMARY KEY (thread_id, group_id)`
138
139	6. `thread_terminal_group_members`
140	- `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
141	- `group_id TEXT NOT NULL`
142	- `terminal_id TEXT NOT NULL`
143	- `position INTEGER NOT NULL`
144	- `PRIMARY KEY (thread_id, group_id, terminal_id)`
145
146	7. `messages`
147	- `id TEXT PRIMARY KEY`
148	- `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
149	- `role TEXT NOT NULL` (`user`/`assistant`)
150	- `text TEXT NOT NULL`
151	- `created_at TEXT NOT NULL`
152	- `streaming INTEGER NOT NULL DEFAULT 0`
153	- `provider_item_id TEXT NULL`
154	- `turn_id TEXT NULL`
155	- `position INTEGER NOT NULL`
156
157	8. `message_attachments`
158	- `message_id TEXT NOT NULL REFERENCES messages(id) ON DELETE CASCADE`
159	- `attachment_id TEXT NOT NULL`
160	- `type TEXT NOT NULL`
161	- `name TEXT NOT NULL`
162	- `mime_type TEXT NOT NULL`
163	- `size_bytes INTEGER NOT NULL`
164	- `position INTEGER NOT NULL`
165	- `PRIMARY KEY (message_id, attachment_id)`
166
167	9. `thread_turn_summaries`
168	- `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
169	- `turn_id TEXT NOT NULL`
170	- `completed_at TEXT NOT NULL`
171	- `status TEXT NULL`
172	- `assistant_message_id TEXT NULL`
173	- `checkpoint_turn_count INTEGER NULL`
174	- `PRIMARY KEY (thread_id, turn_id)`
175
176	10. `thread_turn_diff_files`
177	- `thread_id TEXT NOT NULL REFERENCES threads(id) ON DELETE CASCADE`
178	- `turn_id TEXT NOT NULL`
179	- `path TEXT NOT NULL`
180	- `kind TEXT NULL`
181	- `additions INTEGER NULL`
182	- `deletions INTEGER NULL`
183	- `PRIMARY KEY (thread_id, turn_id, path)`
184
185	11. `provider_events`
186	- `seq INTEGER PRIMARY KEY AUTOINCREMENT`	109	- `seq INTEGER PRIMARY KEY AUTOINCREMENT`
187	- `id TEXT NOT NULL UNIQUE`	110	- `id TEXT NOT NULL UNIQUE`
188	- `session_id TEXT NOT NULL`	111	- `session_id TEXT NOT NULL`
@@ -199,27 +122,43 @@ Use strict schema + migrations (e.g. `PRAGMA user_version` or migration table), · 10 unmodified lines
199	- `payload_json TEXT NULL`	122	- `payload_json TEXT NULL`
200	- `created_at TEXT NOT NULL`	123	- `created_at TEXT NOT NULL`
201		124
202	12. `state_events`	125	3. `state_events`
203	- `seq INTEGER PRIMARY KEY AUTOINCREMENT`	126	- `seq INTEGER PRIMARY KEY AUTOINCREMENT`
204	- `event_type TEXT NOT NULL` (e.g. `project.upsert`, `thread.delete`, `message.upsert`)	127	- `event_type TEXT NOT NULL` (e.g. `project.upsert`, `thread.delete`, `message.upsert`)
205	- `entity_id TEXT NOT NULL`	128	- `entity_id TEXT NOT NULL`
206	- `payload_json TEXT NOT NULL`	129	- `payload_json TEXT NOT NULL`
207	- `created_at TEXT NOT NULL`	130	- `created_at TEXT NOT NULL`
208		131
209	13. `metadata`	132	4. `metadata`
210	- `key TEXT PRIMARY KEY`	133	- `key TEXT PRIMARY KEY`
211	- `value_json TEXT NOT NULL`	134	- `value_json TEXT NOT NULL`
212	- for one-time migrations and import markers	135	- for one-time migrations/import markers
213		136
214	### Key indexes	137	### Envelope indexes
215	- `projects(cwd)`	138	- `documents(kind)`
216	- `threads(project_id, created_at DESC)`	139	- `documents(project_id, kind)`
217	- `messages(thread_id, position)`	140	- `documents(thread_id, kind, sort_key)`
		141	- `documents(kind, updated_at DESC)`
218	- `provider_events(session_id, seq)`	142	- `provider_events(session_id, seq)`
219	- `provider_events(thread_id, seq)`	143	- `provider_events(thread_id, seq)`
220	- `thread_turn_summaries(thread_id, completed_at DESC)`
221	- `state_events(seq)`	144	- `state_events(seq)`
222		145
		146	### Document conventions
		147	1. Validate each `kind` with Zod before writes and after reads.
		148	2. Keep envelope fields stable (`kind`, `project_id`, `thread_id`, `sort_key`) for fast list/filter queries.
		149	3. Put fast-changing/evolving structure in `data_json`:
		150	- project: name/cwd/model/expanded/scripts
		151	- thread: title/model/codex thread id/terminal layout/branch/worktree
		152	- message: role/text/attachments/streaming/provider ids
		153	- turn_summary: completed/status/checkpoint turn count/file +/- summary
		154	4. Use `schema_version` per document to support incremental document migrations without global table churn.
		155
		156	### Normalization runway (later, optional)
		157	When model shapes stabilize, we can materialize normalized tables from `documents` without changing the external protocol:
		158	1. Add typed read models (`projects_v2`, `messages_v2`, etc.).
		159	2. Backfill from `documents`.
		160	3. Switch read paths gradually.
		161
223	## Streaming Protocol and Synchronization	162	## Streaming Protocol and Synchronization
224	Keep current WS request/response envelope and push model. Extend with state snapshot + patch stream.	163	Keep current WS request/response envelope and push model. Extend with state snapshot + patch stream.
225		164
@@ -416,14 +355,14 @@ Optional future mode (not required for this revamp): · 190 unmodified lines
416	2. Add migration boot in `apps/server/src/index.ts`.	355	2. Add migration boot in `apps/server/src/index.ts`.
417	3. Update default state path to `~/.t3/state.sqlite` (with env override).	356	3. Update default state path to `~/.t3/state.sqlite` (with env override).
418		357
419	### Phase 1: Projects + threads in DB	358	### Phase 1: Document store cut-in (projects + threads first)
420	1. Replace `ProjectRegistry` JSON backend with DB-backed repository.	359	1. Replace `ProjectRegistry` JSON backend with DB-backed document repository.
421	2. Add `threads.*` methods to server routes.	360	2. Add `threads.*` methods to server routes.
422	3. Keep web reducer but stop writing to localStorage for these entities.	361	3. Keep web reducer but stop writing to localStorage for these entities.
423		362
424	### Phase 2: Message + turn persistence	363	### Phase 2: Message + turn document persistence
425	1. Ingest provider events into DB (`provider_events`).	364	1. Ingest provider events into DB (`provider_events`).
426	2. Project events into `messages` and `thread_turn_*`.	365	2. Project events into `documents` rows for `message` and `turn_summary` kinds.
427	3. Add `state.event` push and `state.catchUp`.	366	3. Add `state.event` push and `state.catchUp`.
428		367
429	### Phase 3: Web client cutover	368	### Phase 3: Web client cutover
@@ -439,6 +378,7 @@ Optional future mode (not required for this revamp): · 9 unmodified lines
439	### Phase 5: Cleanup + compatibility	378	### Phase 5: Cleanup + compatibility
440	1. Remove deprecated localStorage/persistence schema paths once migration is stable.	379	1. Remove deprecated localStorage/persistence schema paths once migration is stable.
441	2. Keep temporary migration fallback for one release window.	380	2. Keep temporary migration fallback for one release window.
		381	3. (Optional) add a follow-up normalization milestone when schemas settle.
442		382
443	## Streaming/Projection Design Notes	383	## Streaming/Projection Design Notes
444	1. Durable append:	384	1. Durable append:
@@ -465,7 +405,7 @@ Backend changes require new tests and runtime validation. · 20 unmodified lines
465		405
466	### Unit tests	406	### Unit tests
467	1. DB schema + migrations	407	1. DB schema + migrations
468	2. Repository CRUD (projects/threads/messages)	408	2. Document repository CRUD (`project`/`thread`/`message`/`turn_summary`)
469	3. Provider event projection logic	409	3. Provider event projection logic
470	4. Checkpoint summary parser (`@pierre/diffs` integration and malformed diff fallback)	410	4. Checkpoint summary parser (`@pierre/diffs` integration and malformed diff fallback)
471	5. Keybindings JSON behavior remains unchanged (read/write + malformed-file handling)	411	5. Keybindings JSON behavior remains unchanged (read/write + malformed-file handling)
@@ -498,10 +438,12 @@ Backend changes require new tests and runtime validation. · 26 unmodified lines
498	- Mitigation: strict schema validation + partial-row rejection + migration markers + backups.	438	- Mitigation: strict schema validation + partial-row rejection + migration markers + backups.
499	5. Risk: checkpoint summary parsing overhead.	439	5. Risk: checkpoint summary parsing overhead.
500	- Mitigation: parse only finalized checkpoint ranges and fallback to lightweight stat extraction.	440	- Mitigation: parse only finalized checkpoint ranges and fallback to lightweight stat extraction.
		441	6. Risk: JSON shape drift across fast iterations.
		442	- Mitigation: per-kind Zod schemas + `schema_version` in documents + migration tests for older payloads.
501		443
502	## Done Criteria	444	## Done Criteria
503	1. No core durable app data depends on renderer localStorage.	445	1. No core durable app data depends on renderer localStorage.
504	2. Projects/threads/messages persist in `~/.t3/state.sqlite`.	446	2. Projects/threads/messages/turn summaries persist in `~/.t3/state.sqlite` via JSON-first documents.
505	3. Checkpoint patch content remains Git-backed; optional summaries are in DB.	447	3. Checkpoint patch content remains Git-backed; optional summaries are in DB.
506	4. Keybindings remain JSON-backed at `~/.t3/keybindings.json`.	448	4. Keybindings remain JSON-backed at `~/.t3/keybindings.json`.
507	5. Theme/editor/last-invoked-script remain browser-scoped (or explicitly migrated in a separate opt-in feature).	449	5. Theme/editor/last-invoked-script remain browser-scoped (or explicitly migrated in a separate opt-in feature).
@@ -513,8 +455,8 @@ Backend changes require new tests and runtime validation. · 5 unmodified lines
513	- `apps/server/src/index.ts` (state db path + bootstrap)	455	- `apps/server/src/index.ts` (state db path + bootstrap)
514	- `apps/server/src/wsServer.ts` (new state/thread routes + push channel)	456	- `apps/server/src/wsServer.ts` (new state/thread routes + push channel)
515	- `apps/server/src/providerManager.ts` (persist event hooks + checkpoint summary writes)	457	- `apps/server/src/providerManager.ts` (persist event hooks + checkpoint summary writes)
516	- `apps/server/src/projectRegistry.ts` (replace or wrap with DB repo)	458	- `apps/server/src/projectRegistry.ts` (replace or wrap with document-backed repo)
517	- new persistence modules (`stateDb`, migrations, repositories)	459	- new persistence modules (`stateDb`, migrations, document repository, persistence service)
518		460
519	### Web	461	### Web
520	- `apps/web/src/store.ts` (remove localStorage core persistence)	462	- `apps/web/src/store.ts` (remove localStorage core persistence)
@@ -530,7 +472,7 @@ Backend changes require new tests and runtime validation. · 9 unmodified lines
530		472
531	## Open Questions	473	## Open Questions
532	1. Should runtime mode (`full-access` vs `approval-required`) remain browser-scoped UX state, or be elevated to machine policy?	474	1. Should runtime mode (`full-access` vs `approval-required`) remain browser-scoped UX state, or be elevated to machine policy?
533	2. Should terminal panel layout/state be in SQLite (cross-device consistency) or preference JSON (human-editable)?	475	2. Should terminal panel layout/state be in SQLite (shared) or stay browser-scoped UX state?
534	3. Do we need provider event retention limits from day one?	476	3. Do we need provider event retention limits from day one?
535	4. Do we keep `.logs/threads/*.events.ndjson` as debug mirror after DB event log is live?	477	4. Do we keep `.logs/threads/*.events.ndjson` as debug mirror after DB event log is live?
536	5. For non-desktop/browser mode, should state still centralize to `~/.t3/state.sqlite` or support workspace-local override by default?	478	5. For non-desktop/browser mode, should state still centralize to `~/.t3/state.sqlite` or support workspace-local override by default?