Run: prompt-workspace-foundation

• branch: claude/pipeline-prompt-workspace-foundation-ln61dv
• pr: #917

Spec: The prompt workspace — ICM-structured runtime prompts with versions

• slug: prompt-workspace-foundation
• personas: Admin, CSM, SDM, Expert, Vendor, Customer (every persona reads assembled prompts downstream); the edit surface belongs to Sustentus staff (console sustentus audience)
• touches: packages/services/src/ai, packages/services/src/db/models, packages/services/src/db/services, packages/services/src/db/migrations, packages/services/src/server, apps/agent/lib/ai/assistant, apps/agent/app/api/chat, apps/agent/AGENTS.md
• complexity: complex

Problem

Every agent prompt on the platform is a compiled TypeScript template literal — apps/agent/lib/ai/assistant/prompt.ts is one 3.3 kB string. Changing a single word of the assistant's instructions is a code change, a @sustentus/services rebuild and a Vercel deploy, which puts the wording of the product's most visible AI surface behind an engineer and a release cycle. That blocks Q2-2026 Objective 3 — Validate Technical Infrastructure & Payout Flow under the Scale the bridge initiative: the agentic layer cannot be validated or tuned at the pace the objective needs while every correction is a deploy. It also compounds — the eight register stubs behind this one would each hardcode more prompt text that later has to be migrated, so the foundation has to land before them, not after.

Proposed change

A platform-scoped prompt workspace stored in MongoDB and shaped as the ICM layers, with a runtime assembler that composes the system prompt per request.

The layers, and who owns each. L0 splits in two: a rails preamble that is code-owned, always composed in, and never editable (tenant isolation, injection defences, output law), plus a staff-editable identity/voice file. L1 routing — which persona has which register, and which contract file it uses — is derived from code and is read-only. L2 is one markdown contract per register, staff-editable. L3 is staff-editable reference material a register names by path (tone of voice, caveat language, domain vocabulary). L4 — the live conversation, tool output, tenant context — is assembled per request and never stored in the workspace.

Storage and versioning. A file is identified by its ICM path (L0/identity.md, L2/registers/assistant.md, L3/tone-of-voice.md) and carries exactly one piece of mutable state: a pointer at its published version. A version is an immutable document — body, author, timestamp — with no update path anywhere in the service layer. Publishing writes a new version and moves the pointer; rollback re-points at an existing version and writes no new body. Two collections, not an array of versions on the file: version history is unbounded and belongs in its own collection.

Assembly. At request time the assembler resolves the published version of each file a register needs and concatenates fixed slots in a fixed order — L0 identity → rails → L2 register contract → L3 references, references in the order the contract names them. That order is today's prompt's own order, which is what makes the seed set reproduce the current instructions byte for byte. Assembly returns both the prompt text and the version set: the map of file path → version id that produced it, plus a stable hash over that map. The assembled text is cached by that hash; because versions are immutable the cache never needs busting. The published-pointer lookup is not cached — one indexed read per request, so a publish takes effect on the very next turn rather than at the end of a TTL.

Validation at publish, not at request. A publish is refused, with a human-readable reason, if the resulting assembled prompt exceeds the ICM band's 8k-token ceiling, if a required section is missing from the file's layer shape, or if the body attempts to override the rails (a documented, deterministic rule set — reserved rails headings and known override phrasings). Below the band's 2k-token floor is a warning, not a refusal: today's prompt is roughly 0.8k tokens, so a hard floor would refuse the seeds themselves.

Seeds and fallback. The repo ships the workspace's current content as markdown seed files next to the assembler, and a migration loads them as the initial published version set. The store overrides seeds wherever a published version exists; where it does not — or where the store is unreachable — the assembler falls back to seeds and logs the degradation. Agents run on seeds, never on nothing.

Consumption. The assistant agent stops importing a prompt constant and takes assembled instructions from the chat route, which already resolves the tenant it needs. The agent itself stays in apps/agent/lib/ai/; the workspace store, assembler and seeds live in packages/services because apps/console (stub 4) edits the same workspace and neither app can import the other. apps/agent/AGENTS.md gains one line scoping its "no prompt in packages/services" rule to compiled agent prompts, which is what the rule was written to prevent.

Acceptance criteria

• Assembling the seed set produces a system prompt byte-identical to today's apps/agent/lib/ai/assistant/prompt.ts instructions string, asserted by a test that compares against the current text.
• The assistant serves turns on an assembled prompt; with an empty store its instructions are the seed assembly, so its behaviour is unchanged from today.
• Publishing a new version of the L0 identity, an L2 register contract or an L3 reference changes the assembled prompt on the next conversation turn, with no deploy and no cache wait.
• Rolling a file back to an earlier version restores that version's assembled prompt exactly, and creates no new version body.
• A version document has no write path after creation — attempting to mutate one through the service layer fails, and history is never rewritten by a publish or a rollback.
• The rails preamble appears in every assembled prompt, and a publish whose body drops or contradicts it is refused (test: a body carrying a reserved rails heading, and one carrying an override phrase, are both rejected).
• A publish that pushes the assembled prompt past the 8k-token ceiling, or that omits a section its layer requires, is refused with a reason a non-engineer can act on. A prompt below the 2k floor publishes with a warning.
• Assembly returns the version set — every file path with the exact version id used, plus a stable hash — so a caller can record which prompt produced a run.
• With the store unreachable, the assistant keeps answering on seeds; the fallback is visible in server logs and produces no user-facing error.
• The seeded workspace is platform-scoped: no tenantId on either collection, and no code path that resolves a workspace file per tenant.

Out of scope

• The console UI — editing, upload, preview, publish and rollback screens are stub 4 (prompt-console-management). This run's edit surface is the seed files and the service API.
• Persisting the version set on a run record. Stub 2 (conversation-persistence) owns threads, messages and run records, and none exist yet; building a competing run document here would collide with it. This run exposes the version set from the assembler — the stamp is written by stub 2/5 on the record they own. The stub's rough criterion is met at the seam, not at the record.
• New registers, new tools, and L1 as data. The seed set carries exactly one L2 contract — the assistant's current one. Stubs 5 and 7 author real registers into this workspace; the register catalogue stays derived from code (OQ-4).
• Per-tenant prompt overrides, git-sync/export of the workspace, four-eyes publish approval, and eval-gated publishing (OQ-2 / OQ-3 / OQ-8 defaults).
• LLM-assisted review of a publish. Validation is deterministic in v1 so a publish is cheap, reproducible and testable.
• Changing which model the assistant runs on, its tools, or its rate limiting.

Open questions

• none. Two decisions were settled at Define and are recorded above: the fixed assembly order places rails second so the seed assembly is byte-identical to today's prompt (rails-first and rails-bookend were both considered and rejected against that regression guarantee); and the store, assembler and seeds live in packages/services with a scoping amendment to apps/agent/AGENTS.md, because stub 4 edits the same workspace from apps/console.
• Non-blocking note: the token count used by the ceiling is a deterministic local estimate — no exact Anthropic tokenizer runs locally. The estimator's ratio is documented at its definition so the ceiling can be re-tuned without a schema change.
• Non-blocking note: the assembled system prompt is a large stable prefix, so a publish invalidates the provider's prompt cache for the next turn. That is the intended trade and needs no mitigation; it is called out so nobody later adds per-request variation into the prefix and defeats caching permanently.

Build notes: prompt-workspace-foundation

• commits: workspace core + tests · store (models, service, migration) · assistant on assembled prompts
• ci: GREEN on 65b6dc7 — Quality Project (format, lint, typecheck, tests) and all four preview builds; the other four Vercel projects are turbo-ignored by this diff

What changed

• packages/services/src/shared/prompt-workspace.ts: the isomorphic vocabulary — the three stored layers (L0/L2/L3), PromptVersionRef, PromptVersionSet, AssembledPrompt. L1 routing and L4 working context are deliberately absent: neither is storable, and naming them here would invite someone to store them.
• packages/services/src/ai/workspace/: the pure half. rails.ts (code-owned preamble), catalogue.ts (L1 routing — which files exist, what shape each keeps, which register reads which), seeds/ (the repo-shipped bodies), compose.ts (slot order, version-set key and hash), validate.ts (publish refusals), tokens.ts (the ICM band). No DB, no process.env, so it exports cleanly from /ai.
• packages/services/src/db/models/prompt-file.ts + prompt-version.ts: the store. A file is identity plus one mutable pointer at its live version; versions are write-once documents in their own collection rather than an array on the file, because version history is unbounded.
• packages/services/src/db/services/prompt-workspace/: resolution (store-first, seed fallback), publish, rollback, history and listing. Exported from /server as promptWorkspaceService.
• packages/services/src/db/migrations/1787961600000-seed-prompt-workspace.ts: the file rows and their indexes.
• apps/agent/lib/ai/assistant/agent.ts + app/api/chat/route.ts: the assistant takes instructions as a parameter and the route assembles them per request. lib/ai/assistant/prompt.ts is deleted — superseding deletes the superseded.
• apps/agent/AGENTS.md: the "no prompt in packages/services" rule is scoped to compiled prompt constants, with the workspace named as the deliberate exception and why.

Acceptance criteria status

• Byte-identical seed assembly — compose.test.ts asserts the seed set composes to legacy-prompt.fixture.ts, a frozen golden copy of the prompt as it shipped. The seeds were cut from the real prompt.ts bytes by script, not transcribed.
• Assistant serves turns on an assembled prompt; an empty store resolves every path to its seed.
• A publish reaches the next turn with no deploy — the published-pointer lookup runs on every request and is deliberately uncached; only the immutable composed text is cached, keyed by the version set, so a publish yields a new key rather than a stale entry.
• Rollback re-points and writes no new body (rollback touches only publishedVersionId).
• Versions have no write path — immutable: true on every path, plus query middleware that refuses updateOne/updateMany/findOneAndUpdate/replaceOne, and no service method that mutates one. See the testing note below.
• Rails always present and un-overridable — present structurally because composePromptText puts them there; publishes carrying a reserved rails heading or an override phrasing are refused (validate.test.ts).
• Over-ceiling and missing-section publishes refused with an actionable reason; a thin prompt publishes with a warning, not a refusal.
• Assembly returns the version set — every path with the exact version id, plus a stable hash.
• Store unreachable ⇒ seeds, logged, no user-facing error (resolveBodies never throws). See the testing note below.
• Platform-scoped — no tenantId on either collection and no per-tenant resolution path.

Notes for Release

• One deliberate departure from the spec's wording. The spec says the migration "loads them as the initial published version set"; the migration instead creates each file row with a null pointer, which resolves to the seed. Publishing the seeds as v1 would freeze them in the store and strand the repo copies — a later prompt improvement in code would never reach production, because the store always wins. With a null pointer an untouched file tracks the repo and the store takes over the moment a human publishes. No acceptance criterion distinguishes the two, and the spec's own sentence "the store overrides seeds wherever a published version exists" describes what was built. Flagging it because it contradicts one sentence I wrote at Define.
• Seeds are TypeScript modules holding markdown, not .md files on disk. TSUP does not copy non-TS assets into dist/ and Next.js does not trace them into the serverless bundle, so a readFileSync seed would work in dev and throw in production — the exact failure the fallback exists to prevent. The bodies are still markdown and still one module per workspace file.
• Two criteria are met by construction but carry no unit test: version immutability and the store-unreachable fallback. Both need a live Mongo to exercise, and only the unit tier is configured — CONVENTIONS.md → Testing forbids improvising the integration tier. Worth a look in review, and worth exercising by hand on the preview.
• The migration must run before publishing works in a given environment. Assembly does not depend on it: with no rows at all, every path resolves to its seed and the assistant is unchanged. Migrations apply on merge to main via db-migrate.yaml.
• apps/agent/AGENTS.md changes in this PR. A convention edit riding with the change that motivates it — worth reading as part of the review rather than skimming as docs.
• This PR's labels are set by hand, and CI will clobber them again on the next push. The labels job projects every run folder under .icm/runs/ onto the PR being pushed, so #917 was left wearing stage:ship persona:csm persona:vendor — another run's labels. Re-running project-labels.sh prompt-workspace-foundation --stage build --pr 917 repairs it. Parked as .icm/intake/triage/pipeline-labels-job-clobbers-pr-labels.md with the job-log evidence and the archival backlog behind it; not absorbed into this diff. Check the label before trusting the board on this PR.
• No new environment variables, so turbo.json → globalEnv is untouched.

Release

• gate: Ready to merge ticked — merge authorised
• ci: GREEN on da6d34d (ci-status.sh), with Migrate preview database passing on the repaired migration — the direct proof the collation fix converges. This record commit is .icm-only on top of it and is re-verified GREEN on the head that merges.
• reviews: code high (3 findings — 2 fixed, 1 parked) · security run — no findings · readiness run — 1 blocker, fixed after CI proved the real failure
• parked: pipeline-labels-job-clobbers-pr-labels.md · prompt-workspace-write-path-uses-degraded-reads.md
• docs: apps/docs technical/packages/services updated · announce: internal

Fixed during Release (all on this diff, none widening it)

• Index/migration lockstep — the one blocker, and it went a round. Both workspace indexes were created without the { locale: "en", strength: 2 } collation schemaPlugin stamps on the models, so the published-pointer lookup on the assistant's hot path could not use them and Mongoose's autoIndex build collided on the index name. The first attempt corrected it fix-forward, on the readiness pass's finding that the original migration had already been applied to preview. CI then failed and showed that premise was wrong: the job log has seed-prompt-workspace ... State is down — its record had been pruned from the shared preview database by another branch's migrate run (Removing migration(s) from database: agent-conversation-indexes), and production has never run it, so it had not durably applied anywhere. The correct fix was therefore to repair the original migration rather than layer a second one over it, which is what shipped: the indexes are dropped-then-created with collation, so it converges whether Mongoose built the collated index first or an uncollated one is still in place. The redundant 1788610000000-prompt-workspace-indexes-collation was deleted.
• The original migration's down destroyed published prompt history. up creates three file rows and no versions; down deleted every promptversions row for those paths — i.e. every prompt a human had published after it ran. Because the migration turned out not to be applied anywhere, this could be fixed in place rather than parked: down now removes only the file rows it created and only while they are still unpublished, and never touches promptversions. Its two dropIndex calls also stopped swallowing every error and now use the house ABSENT regex.
• publish() skipped validation entirely for a catalogued file no register composes yet — a file can land ahead of its register, and that path stored a body with no empty, section or rails check. It now validates against the body alone when there is no reader.
• A file pointing at a missing version fell back to its seed silently — it now logs, so the console cannot report a published version the assistant is not running on.

Not fixed, and why

• The two parked stubs are the write-path/degraded-read gap and the CI labels job. Neither is a stop class, and the labels job is pipeline infrastructure this diff cannot reach.
• publish() still performs no permission check and takes publishedBy as a caller-supplied string. Nothing exposes it — the only caller in the repo is the chat route's read path — so there is nothing to exploit today. prompt-console-management must gate its route on the sustentus audience and derive publishedBy from the session; its stub already carries that requirement.

A note for whoever hits this next

The shared preview database is not a reliable record of what has been applied: a migrate run from any branch prunes the records of migrations whose files that branch does not have. So "the migrate-preview job passed on this PR" does not mean a migration is durably applied, and packages/services/AGENTS.md's never-edit rule has to be judged on whether the migration has reached production, not on a green preview job. Reading State is down in the job log is the check that settles it.