diff --git a/docs/plans/mbrian-backend-migration.md b/docs/plans/mbrian-backend-migration.md index ad7084b..637a6b9 100644 --- a/docs/plans/mbrian-backend-migration.md +++ b/docs/plans/mbrian-backend-migration.md @@ -1,12 +1,20 @@ # mBrian-as-backend migration — Phase 6 design -**Status**: Phase A design (this doc). +**Status**: Phase A design — re-baselined against live mBrian schema (2026-05-29). **Branch**: `mai/kahn/phase-6a-mbrian-design`. **Author**: kahn (inventor), 2026-05-29. **Source decision** (m, issue m/projax#5, 12:43 2026-05-29): Option A — full backend migration. *"I think we need the project-management element inside of mBrian for it to be the complete 2nd Brain experience. The data itself is not too important yet."* +**m's overriding directive** (2026-05-29 via head): *"keep the database simple so it remains easily modifiable."* + **Constraint**: data-loss tolerant on the 47 current `projax.items`. +**m's answers on §10 (2026-05-29)**: every inventor pick confirmed. + +> Q1=reuse 'project' / Q2=(b) handler bridge / Q3=(a) clients projax-side / Q4=(a) file Gitea on m/mBrian via otto/head — m: *"mbrian must own the migration"* / Q5=(a) views stay projax-resident / Q6=(a) per-user slug / Q7=(a) hard-cut / Q8=(a) tags in metadata / Q9=(a) projax-side cycle detection / Q10=(a) keep projax MCP via adapter / Q11=keep `projax_origin` audit metadata. + +**Re-baseline note**: §3's original ask was built off a stale `db/001_initial_schema.sql` read. Head verified the live mBrian schema after m's answers. Three of the six asks (MB-A, MB-C, MB-D) turned out already-satisfied — `edges.metadata` exists since `db/010_flexsiebels_compat.sql`, `'project'` type exists since `db/033`, the per-user slug unique index ships in `db/001`. The remaining mBrian-side artifact is small. §3 + §8 now reflect that. The big shift: **mBrian owns the one-shot data-migration script** — that's what "mbrian must own the migration" means — while projax owns the read+write rewiring on its own side afterward. + --- ## §1 — Diagnosis @@ -32,7 +40,7 @@ End-state contract: | projax column | mBrian destination | notes | |---|---|---| | `id` (uuid) | `nodes.id` | new uuids on migration; legacy ids never round-trip | -| `kind` (text[]) | `nodes.type` | direct shape match; projax `'project'` becomes mBrian `'project'` (already exists per mig 030); add `'area'` if missing | +| `kind` (text[]) | `nodes.type` | direct shape match; projax `'project'` becomes mBrian `'project'` (already in live schema, mig 033). **Areas keep `type=['project']` + `metadata.projax.kind='area'`** — per m's "keep the database simple" directive, no new mBrian type. Zero DDL. | | `title` | `nodes.title` | 1:1 | | `slug` | `nodes.slug` | mBrian = unique per user; projax = unique per parent — see §2.1 | | `paths` (text[]) | derived from `child_of` edges + `nodes.path` cache | DAG resolution via edge walk; see §2.2 | @@ -52,11 +60,13 @@ End-state contract: | `updated_at` | `nodes.updated_at` | trigger-maintained on both sides | | `deleted_at` | `nodes.deleted_at` | 1:1 | -### §2.1 — Slug uniqueness +### §2.1 — Slug uniqueness (settled) -projax enforces slug uniqueness **per parent** (a `paliad` slug can exist under both `dev` and `work`). mBrian enforces slug uniqueness **per user** (one `paliad` total). For a multi-parent item like `paliad` living under both `dev` and `work`, mBrian today only allows one `paliad` node — which is actually what we want: a single canonical node connected to both parents via `child_of` edges. The DAG-as-multiple-paths view is a render-time concept; the storage is one node. +projax today enforces slug uniqueness **per parent**. mBrian's live schema has `CREATE UNIQUE INDEX idx_nodes_slug ON mbrian.nodes (user_id, slug)` — uniqueness **per user**. Per m's Q6=(a), projax adopts mBrian's model: one `paliad` node total, connected to both `dev` and `work` via two `child_of` edges. The DAG-as-multiple-paths view is a render-time concept; the storage is one node. -The projax handlers' itemwrite validator (Phase 5c) loses its per-parent slug rule and gains an uniqueness-per-user check. This is **stricter** — m can't have two different "paliad" projects under different roots. Flag for Q6. +projax handlers' itemwrite validator (Phase 5c) loses its per-parent slug rule, gains a per-user check (against the projax-managed subset of nodes). This is **stricter** — m can't have two different "paliad" projects under different roots. Settled per m's answer. + +**Pre-migration dedup**: the 47-item migration script (which lives mBrian-side, see §3+§7) scans for slug collisions across the projax dataset and folds collisions into one node with multiple `child_of` edges. Skip-with-log on anything weirder. ### §2.2 — paths array vs single path @@ -91,29 +101,45 @@ mBrian's `edges` table today has no `metadata jsonb` column — only `rel`, `not --- -## §3 — mBrian-side requirements +## §3 — mBrian-side requirements (re-baselined against live schema) -Schema fragments mBrian needs to register. Each is a separate migration on the mBrian side, written by mBrian's coder. These are the cross-repo asks: +Head verified the live mBrian schema after m's answers. Three of the original six asks turned out already-satisfied. What's actually needed reduces to one [schema] convention node + ownership of the one-shot data-migration script. Per m's Q4=(a), this lands as a Gitea issue on `m/mBrian` with the "blocks projax phase 6" tag; head files it. -**MB-A — Add `edges.metadata jsonb NOT NULL DEFAULT '{}'`** (per §2.2). Backfill is no-op; new column starts empty. Optional GIN index if projax queries by metadata content (not in v1 — projax queries by `rel` only). +### Already satisfied (no DDL needed) -**MB-B — Register projax-specific edge relations.** mBrian has no enum on `edges.rel`; new values just appear. Document the projax-namespaced rels (`projax-caldav-list`, `projax-gitea-repo`, `projax-gitea-issue`, `projax-mai-project`, `projax-url`, `projax-doc`) in a schema-node so mBrian's tooling knows they belong to projax. Concrete deliverable: a `[schema]` node `projax-edge-relations` under a `[topic]` hub `projax-integration` (created same migration). +| original ask | live-schema status | +|---|---| +| MB-A — `edges.metadata jsonb` column | **Already exists** — added in `db/010_flexsiebels_compat.sql`: `ALTER TABLE mbrian.edges ADD COLUMN IF NOT EXISTS metadata jsonb NOT NULL DEFAULT '{}'` plus GIN `idx_edges_metadata`. Already used by mig 039/040. projax link payloads land here directly. | +| MB-C — `'project'` type registration | **Already exists** — confirmed in `db/033` + inbox tests. m's Q1=(a) reuses it. | +| MB-C — `'area'` type registration | **NOT needed** — per m's "keep the database simple," areas reuse `type=['project']` with `metadata.projax.kind='area'`. Zero DDL. | +| MB-D — per-user slug uniqueness | **Already enforced** — `CREATE UNIQUE INDEX idx_nodes_slug ON mbrian.nodes (user_id, slug)` in `db/001`. Handles the bulk migration as-is, modulo the pre-write dedup pass in the script (§7). | +| MB-E — read MCP coverage | **Confirmed** by head — type-array filter, edge query by `rel` + source/target, FTS search all present in mBrian's MCP today. Optional bulk "node + outbound edges" endpoint may improve adapter perf, but v1 ships without it. | +| MB-F — write MCP coverage | **Confirmed** by head — create_node, update_node, soft-delete, create_edge, delete_edge all present. | -**MB-C — Optional: register projax types as known types.** mBrian's `'project'` type already exists. Add `'area'` if missing (projax's top-level area concept). Document the projax type set in another `[schema]` node so future readers know `'mai-managed'` means "this projax node mirrors a mai.projects row." +### Remaining mBrian-side artifact -**MB-D — Confirm no slug-uniqueness blockers.** projax migration creates ~47 nodes with potentially-colliding slugs (e.g. two `paliad`-rooted DAG paths today are ONE node post-migration; need to dedupe pre-write). mBrian-head to confirm the per-user unique index doesn't choke on bulk insert ordering. +**MB-B — projax-integration `[schema]` convention node.** One new mBrian node, no DDL. Lives under a new `[topic]` hub `projax-integration`. Documents: -**MB-E — Read-only API surface.** projax read-path (§4) calls mBrian. We need: -- A way to filter nodes by `type` array containment (mBrian already exposes via `list_nodes(type='...')`). -- Edge query by `rel` + `source_id` or `target_id`. -- Trigram / FTS search across title + content_md (mBrian already has this). -- Optionally: a bulk endpoint that returns nodes + their outbound edges in one call (projax's tree-render path needs ~all nodes + all edges). +1. The projax edge relations: `child_of` (already in use everywhere), `projax-caldav-list`, `projax-gitea-repo`, `projax-gitea-issue`, `projax-mai-project`, `projax-url`, `projax-doc`. Each entry: rel name + the metadata jsonb shape (e.g. `projax-caldav-list` carries `{url: text}`). +2. The projax type usage: `'project'` for both projects and areas; `metadata.projax.kind` distinguishes (`area` vs default `project`). `'mai-managed'` as a co-type marker for nodes mirroring `mai.projects` rows. +3. The projax metadata shape: `metadata.projax.{status, tags, management, public, timeline_exclude, start_time, end_time, kind}` — the subset of projax columns that don't have a first-class mBrian counterpart. +4. A pointer to `projax_origin` audit metadata (set per migrated node, per m's Q11=keep). -Confirm MCP surface coverage. If anything's missing, mBrian-coder adds it. +mBrian-side coder writes this node by creating it via mBrian's editor or MCP. No migration file needed. -**MB-F — Write API surface.** projax write-path (§5) calls mBrian. We need create_node, update_node, soft-delete, create_edge, delete_edge — all already exist in mBrian's MCP. +### mBrian owns the data-migration script -Cross-repo coordination shape: this lands as a Gitea issue on `m/mBrian` repo, batched as one ticket with sub-tasks for A–F. The issue links back to this plan + projax/#5. +Per m's directive "mbrian must own the migration," the one-shot script that creates the 47 nodes + their edges lives in `m/mBrian` (likely `scripts/migrate-from-projax.ts` or similar — mBrian's stack picks). projax-side provides: + +- A frozen snapshot of `projax.items` + `projax.item_links` rows (CSV or JSON dump produced by a projax-side helper). +- The mapping rules from §2 + §2.2 in a form mBrian-side can implement against (this plan doc is the canonical source). +- A spot-check checklist (5 representative items) for post-migration validation. + +The script's blast radius lives on mBrian's side; projax-side blocks on its successful run before slice C kicks off. + +### Cross-repo coordination shape + +One Gitea issue on `m/mBrian` (filed by head), tagged "blocks projax phase 6". The issue body covers MB-B + script ownership + the snapshot-handoff protocol. Body draft delivered to head with this re-baseline (see Phase A workflow §14). --- @@ -199,73 +225,99 @@ This is **Q2** for m. --- -## §7 — Migration mechanics +## §7 — Migration mechanics (mBrian-owned) -Per m's loss-tolerance signal: hard-cut. One script, run once, with a clear blast radius. +Per m's Q7=(a) hard-cut + Q4=(a) "mbrian must own the migration": the one-shot script lives in `m/mBrian`. projax-side provides the input snapshot + the rules in this doc; mBrian-side owns the execution. -Script outline (Go, lives in `cmd/migrate-mbrian/main.go`): +### projax-side input snapshot -1. Connect to msupabase as `postgres`. -2. Read every row from `projax.items` where `deleted_at IS NULL`. -3. For each row: - a. Generate new mBrian uuid (or preserve old uuid — projax uuids don't collide with mBrian's; we can preserve, but the migration script picks). - b. INSERT into `mbrian.nodes` with type/title/slug/aliases/metadata mapped per §2. Add `metadata.projax_origin: ` so a future audit can reconcile. - c. For each `parent_id`, INSERT `mbrian.edges (source=new_node_id, target=parent_new_id, rel='child_of')`. Parent uuids resolved via a two-pass walk: first pass creates all nodes, second pass writes edges. -4. For each `projax.item_links` row: - a. Translate `ref_type` → mBrian edge `rel` per §2.2 table. - b. INSERT `mbrian.edges` with `metadata` carrying the structured payload. -5. For each `projax.views` row (5j paliad-shape views): see Q5. -6. Smoke check: count(mbrian.nodes WHERE metadata->>'projax_origin' is not null) == count(projax.items WHERE deleted_at IS NULL). -7. DON'T drop projax.items + item_links in the same migration. Drop happens in slice E after stable read+write. +A helper command in `cmd/projax-snapshot/main.go` produces a `projax_snapshot.json` containing every live `projax.items` row + every `projax.item_links` row, shaped for direct consumption by the mBrian-side script. One file, deterministic, round-trippable. Ships in slice 0 (the snapshot handoff, see §8). -Idempotency: the script checks `metadata.projax_origin` on each insert to avoid duplicates on re-run. +### mBrian-side script outline (for the m/mBrian issue body) -Lossy bits (acceptable per m's stance): the projax `paths` array isn't preserved — it's recomputed from edges. The Phase 1.5 mai.projects mirror rows: the bridge worker handles re-sync after migration (Q2). +1. Load `projax_snapshot.json`. +2. Two-pass: pass 1 creates every node; pass 2 writes every edge (parent edges + item_links → projax-* edges). +3. For each item: + a. New mBrian uuid OR preserve the projax uuid (mBrian-side picks; either works given m's Q11 audit metadata is the durable reference). + b. INSERT into `mbrian.nodes` with `type=['project']` (or `['project']` + co-type per `kind`), `title`, `slug`, `aliases`, `metadata={projax: {...}, projax_origin: }`. + c. Where projax had multiple paths (same node under multiple parents), DEDUPE by slug — one node, multiple `child_of` edges. +4. For each parent edge: INSERT `mbrian.edges (source=new_id, target=parent_new_id, rel='child_of')`. +5. For each item_links row: INSERT `mbrian.edges` with `rel='projax-'` and `metadata` carrying the structured payload per §2.2. +6. For projax.views (5j): NOT migrated — per m's Q5=(a), the views table stays projax-resident. +7. Smoke check: count(mbrian.nodes WHERE metadata->>'projax_origin' is not null) == count(items in snapshot). +8. Hand off to projax with the new uuid map (`{old_uuid: new_uuid}`) so projax-side caches can warm. + +### Idempotency + +Pre-flight: the script checks `metadata.projax_origin` and skips already-migrated origins on re-run. m can re-run safely if the script aborts mid-way. + +### Lossy bits (acceptable per m's stance) + +- `paths text[]` array is not preserved — projax-side adapter recomputes from edges per §4. +- mai.projects mirror rows: per Q2=(b), a handler-layer bridge worker re-syncs after migration; the Phase 1.5 trigger pair stays disabled. + +### Blast-radius containment + +mBrian-side runs the script with triggers paused, smoke-checks the count + spot-checks the 5 representative items in projax's checklist, then commits + signals projax-side to start slice C (read-path). --- -## §8 — Implementation slicing +## §8 — Implementation slicing (re-baselined) -Slices A–F, with hard cross-repo coordination on A. Each slice independently shippable. +Six slices. The big shift from the original draft: the mBrian-side ask compresses to one [schema] convention node + one migration script (both mBrian-owned per m's Q4). Slice 0 is a small projax-side helper that ships the snapshot. The hard gate is the migration landing — projax-side B reads it as the trigger to start. -- **A. mBrian schema-fragment** — mBrian-side. Adds `edges.metadata`, registers projax edge relations + types in schema-nodes, confirms read+write MCP coverage. Lands as a Gitea issue on `m/mBrian`, coordinated through §9. +- **0. projax-side snapshot helper** — `cmd/projax-snapshot/main.go`. Dumps live `projax.items` + `projax.item_links` to `projax_snapshot.json`. Ships first; minimal risk; deliverable mBrian needs. -- **B. Data migration script** — projax-side. `cmd/migrate-mbrian/main.go`. Runs once against the 47 items + their links + the 5j views (per Q5). Test on a scratch mBrian dataset before the real migration. +- **A. mBrian-side: [schema] convention node + data-migration script** — m/mBrian owns. The [schema] node lives under a new `[topic]` hub `projax-integration`. The script consumes the snapshot from slice 0 and writes 47ish nodes + their edges per §7. mBrian-side post-flight: smoke-check count + spot-check 5 items per the projax checklist. -- **C. Read-path replacement** — projax-side. `store/` package rewired to mBrian. The Item struct stays; method bodies rewrite. All UI + aggregator tests stay green (they only see Item). Adapter caches a per-request snapshot to avoid N+1 mBrian calls. +- **B. projax-side read-path adapter** — projax-side. `store/` package rewired against mBrian's MCP / SQL surface. The `Item` struct stays; method bodies rewrite. All UI + aggregator tests stay green (they only see Item shape). Per-request snapshot cache to avoid N+1 calls. Reads-only soak before slice C. -- **D. Write-path replacement** — projax-side. Every handler + MCP write rewires. itemwrite validator updates for slug-uniqueness semantics. +- **C. projax-side write-path** — projax-side. Every handler + MCP write rewires through the adapter to mBrian. itemwrite validator updates for the per-user slug rule (Q6). Cycle detection on the in-memory closure (Q9). -- **E. Drop projax tables** — projax-side. After stable read+write on mBrian for one shift, drop `projax.items` + `projax.item_links`. Migration `0018_drop_projax_items.sql`. +- **D. mai.projects bridge worker** — projax-side (Q2=(b)). Disable the Phase 1.5 trigger pair; ship a small worker that observes mai.projects writes + reflects them into mBrian, and vice versa. Decoupled, killable. -- **F. Integrations disposition** — depending on Q2's answer. If (b), build the bridge worker. If (c), coordinate with mai-side to migrate workers/tasks FKs. +- **E. Drop `projax.items` + `projax.item_links`** — projax-side. Migration `0018_drop_projax_items.sql`. Triggers off after one shift's stable read+write soak on mBrian. `projax.views` stays (Q5). Dependency graph: ``` -A ──→ B ──→ C ──→ D ──→ E - ↘ F (parallel after D) +0 (projax snapshot) ──→ A (mBrian [schema] node + migration script run) + │ + ▼ + B (projax read-path) ──→ C (projax write-path) + │ + ├──→ D (mai bridge worker) + ▼ + E (drop projax tables) ``` -A is the gate. C and D can ship together if the test surface stays green; otherwise C ships first (read-path) and D follows after one shift's worth of read-only soak. +Slice 0 unblocks A. A is mBrian-owned and the hard gate for everything else. B → C can ship together if green; otherwise B-first soak. + +CalDAV / Gitea integrations stay where they are (Q3=(a)) — no slice F needed in the original sense. --- -## §9 — Cross-repo coordination +## §9 — Cross-repo coordination (settled) -mBrian is m's actual second brain. Schema changes carry blast radius. The protocol: +Per m's Q4=(a) + his words *"mbrian must own the migration"*: -1. **mBrian's head identity**. Looking at the m/mBrian repo: m manages it directly today (no mBrian/head worker registered in mai). Per global Channel Routing rule (only `otto/head` writes to m directly), the projax coordination request routes through `otto/head` who relays to m and back. Confirm with **Q4**. +1. **Protocol**: file a Gitea issue on `m/mBrian` with "blocks projax phase 6" tag. Routed via otto/head per global Channel Routing. Head files it; kahn drafts the body. -2. **The schema-fragment ask**. One delegation message to head with the §3 MB-A through MB-F items batched. Head decides whether to: file as an issue on `m/mBrian` directly, route through otto/head for m to dispatch a mBrian worker, or coordinate paired implementation. +2. **Ownership split**: + - mBrian-side owns: the `[schema]` convention node (MB-B) + the one-shot data-migration script. + - projax-side owns: the snapshot helper (slice 0), the read-path adapter (slice B), the write-path (slice C), the mai bridge (slice D), the table drop (slice E). -3. **Sequencing**. mBrian-side A must land + deploy before projax-side B can run. Recommend filing the schema request as `m/mBrian` issue with a clear "blocks projax phase 6" tag. +3. **Sequencing**: slice 0 produces the snapshot → mBrian-side A consumes it + runs the migration → mBrian-side signals back → projax-side starts B. The Gitea issue is the durable trace; the delegation reply chain is the real-time signal. -4. **Design-doc sharing**. This plan stays in `m/projax`. m/mBrian gets a pointer issue with the relevant §2+§3 excerpts. +4. **Design-doc sharing**: this plan stays in `m/projax`. The m/mBrian issue body (drafted alongside this re-baseline, delivered to head) excerpts §2 (schema mapping), §3 (the one [schema] node ask), §7 (the script outline), and the spot-check checklist. --- -## §10 — Open questions for head delegation +## §10 — Open questions (all answered 2026-05-29) + +All 11 questions resolved. m confirmed every inventor pick. Section retained as the historical record + so a future hand can audit the decision rationale. + + The 8 from issue #5 plus what surfaced during this survey. @@ -375,6 +427,8 @@ Per the migration script, every migrated node carries `metadata.projax_origin = ## §14 — Status -- **Phase A (this doc)**: drafted by kahn, 2026-05-29. Awaiting m's answers on §10 via head delegation, AND cross-repo coordination via head with mBrian. -- **Phase B (coder)**: blocked on (1) m's sign-off on §10 + (2) mBrian-side schema-fragment landed (slice A complete + deployed). -- **No code changes** in this branch beyond this doc. Slice A is mBrian-side; projax-side slices B–F wait on it. +- **Phase A (this doc)**: drafted by kahn 2026-05-29, re-baselined same day against live mBrian schema after m's 11 answers landed. All §10 questions resolved. +- **m/mBrian Gitea issue**: body drafted; head files it under "blocks projax phase 6" tag. +- **Phase B (projax-side coder)**: blocked on (1) slice 0 snapshot helper ships + (2) mBrian-side migration runs + signals back. NO coder flip yet. +- **Slice 0 (projax-side snapshot helper)**: scoped, not yet built. Smallest first-step on projax-side; ready when head greenlights. +- **No code changes** in this branch beyond this doc.