Run: preview-db-wiring

• branch: claude/preview-db-wiring-m7bk03
• pr: #799

Spec: Preview DB wiring — real secrets, loud failures, on-demand catch-up

• slug: preview-db-wiring
• personas: Admin
• touches: .github/workflows/db-migrate.yaml, apps/docs
• complexity: standard
• demo: none

Problem

The migrate-preview job has passed green on every PR while migrating nothing. The GitHub preview environment carries no MONGODB_URI/MONGODB_DATABASE_NAME secrets, so both resolve empty, the production-DB guard passes vacuously, and the run step's exit 0 skip path (.github/workflows/db-migrate.yaml:92–98) turns "not configured" into a green check. Meanwhile the Vercel preview scope points at a real preview database CI never touches, so preview URLs run new code against an unmigrated database — the reported symptom of "the changes aren't there on the preview". A green check at merge time therefore proves nothing, which is exactly what Refine the Bridge / 2026-Q2 Objective 3 (Validate Technical Infrastructure & Payout Flow) needs it to prove: this is stub 1 of 3, the wiring the baseline seed (stub 2) and the preview demo storyline (stub 3) both stand on.

Proposed change

Wire the truth, and make a lie impossible:

• Point the GitHub preview environment at the database Vercel preview already uses — configure MONGODB_URI and MONGODB_DATABASE_NAME on that environment from the value read out of the Vercel preview scope's env settings. No second preview database is minted. Same Atlas cluster as production, distinct dbName. This is a human step in the GitHub and Vercel UIs and it happens before this PR merges (see Sequencing).
• Silent skip becomes a red failure. In migrate-preview, an empty MONGODB_URI — or an empty MONGODB_DATABASE_NAME, without which the guard is half-blind and the summary can't name a target — fails the job with an actionable ::error:: pointing at the preview environment's secrets. migrate-production's own skip path is untouched.
• Say what was migrated. The job appends the resolved database name (never the URI, never any fragment of it) and the migration outcome to $GITHUB_STEP_SUMMARY, so a green check is auditable at a glance without opening logs.
• On-demand catch-up. workflow_dispatch gains a target choice input (preview | production, default preview); each job gates on it, so a manual run migrates one database, not both. A preview DB that lagged behind — after app-only PRs, which the path filter skips — is caught up by dispatching the workflow, with no dummy commit.
• Retire the transitional comment. The guard's comment block (db-migrate.yaml:78–84) describes the no-secrets fallback as the current state; once the secrets exist it is rewritten to describe the guard as the standing denylist it becomes.
• Runbook in apps/docs (technical › development › database / ci-cd): where the preview secrets live, that they must match the Vercel preview scope, how to rotate them, and how to trigger a catch-up dispatch. Written via docs-sync at Ship, in this PR.

Sequencing (why order matters here)

The loud-fail change merged before the secrets exist would turn every packages/services/** PR red. So: configure the secrets first, confirm them with a dispatch run or on this PR's own migrate-preview check, and only then tick Ready to merge. The PR description records that the secrets were verified present.

Acceptance criteria

• The GitHub preview environment has MONGODB_URI + MONGODB_DATABASE_NAME set, and the dbName reported by migrate-preview matches the database the Vercel preview scope's MONGODB_URI targets.
• On a PR touching packages/services/**, migrate-preview connects to the preview database and applies (or confirms up to date) the migrations — no skip path taken.
• The job summary for that run names the resolved database and reports the migration outcome, and contains no connection URI or credential fragment.
• With MONGODB_URI or MONGODB_DATABASE_NAME absent, migrate-preview fails red with a message naming the preview environment secrets — never green.
• The production-DB guard still aborts when the resolved target is sustentus-prod (by dbName or by URI match).
• workflow_dispatch with no input runs migrate-preview only; target: production runs migrate-production only, still behind its required-reviewer environment gate.
• The pull_request path filter is unchanged, and migrate-production's push-to-main behaviour is unchanged.
• A preview-database runbook page exists in apps/docs covering secret location, the Vercel-parity requirement, rotation, and catch-up dispatch.

Out of scope

• Seeding of any kind — the baseline seed is stub 2, the demo storyline stub 3.
• Ordering the Vercel preview build after the migration job — eventual consistency accepted at the cut; the two may race.
• Ephemeral per-PR databases, and periodic reset from a sanitized production snapshot — both declined at the cut.
• A production-parity merge check (db:migrate status against production) — offered, declined.
• Widening the pull_request path filter; app-only PRs still skip migrations, and the dispatch run is the answer to a lagging preview DB.
• migrate-production's skip-on-empty-URI path, its required-reviewer gate, and the migration engine's prune/autosync behaviour.
• DEMO_TENANT_CLERK_ORG_ID for migrate-preview — stub 3 owns the preview demo tenant and revisits it.

Open questions

• none. One decision was taken here rather than in the stub: the dispatch target input defaults to preview, which narrows migrate-production's bare dispatch (it now needs target: production). The stub's "dispatch behaviour unchanged" is met in substance — production is still dispatchable, behind the same reviewer gate — but not literally for an input-less run. Deliberate: the safe target is the default.

Build notes: preview-db-wiring

• commits: feat: preview-db-wiring — loud preview migrations, auditable summary, dispatch target
• demo: none (dev-infra; Design was skipped)

What changed

Only .github/workflows/db-migrate.yaml — no application or package code is touched.

• workflow_dispatch gains a target choice input (preview | production, default preview). migrate-preview's if now admits a dispatch that asked for preview; the pull_request arm keeps its head-repo guard verbatim, so forked PRs still never reach the secrets. migrate-production's if takes the matching target == 'production' arm, leaving its push-to-main behaviour untouched. One manual run therefore migrates exactly one database — a preview catch-up never queues a production deployment for reviewer approval.
• New preflight step, Require the preview environment secrets: an empty MONGODB_URI or MONGODB_DATABASE_NAME now exits 1 with an ::error:: naming the environment, the settings path, and the Vercel-parity requirement. This replaces the exit 0 skip that made a missing secret indistinguishable from a successful migration.
• Preflights run before checkout/install. Neither check needs the repository, and a misconfigured environment now fails in seconds instead of after a dependency install.
• Guard comment rewritten. It described a transitional state (no preview secrets, repo-level fallback to production). With the secrets required above, it is a standing denylist; the logic is unchanged, including the URI substring match.
• Job summary. Run migrations tees db:migrate up, writes the resolved database name, the outcome, and a collapsed output block to $GITHUB_STEP_SUMMARY, then re-raises a non-zero exit. set -o pipefail is set explicitly — without it the | tee would mask a failing migration as success. The embedded output is passed through a sed scrub that redacts any mongodb:// / mongodb+srv:// string, so a driver error quoting the URI cannot leak credentials into the summary.

migrate-production is otherwise untouched: its own skip-on-empty-URI path, its environment: production reviewer gate, and DEMO_TENANT_CLERK_ORG_ID all stand as they were.

Acceptance criteria status

• Preview environment secrets set, matching Vercel's preview scope — the set half is already true and was not true when the stub was cut: see the finding below. The matching half is a human read of two dashboards and is the only part still open.
• migrate-preview connects and migrates on a packages/services/** PR — demonstrated on this PR's own run (job Migrate preview database, run 31694115418): both secrets resolved non-empty, the guard passed on a non-production target, and db:migrate up reported "There are no pending migrations" with all 21 migrations listed up:. No skip path taken.
• Job summary names the database and reports the outcome, with no URI — implemented, but the first attempt did not actually satisfy it and was corrected. See "Reporting the database name" below; the criterion now depends on one dashboard change.
• Missing secret fails red — Require the preview environment secrets exits 1 with an actionable ::error::. Not exercised live (the secrets resolve), so this is verified by inspection, not by a red check. Verify may re-confirm it by temporarily unsetting.
• Production guard still refuses sustentus-prod — logic unchanged (dbName equality plus URI substring match); only its comment was rewritten.
• Dispatch targets one database — bare dispatch → migrate-preview only; target: production → migrate-production only, still behind required reviewers.
• Path filter and push-to-main unchanged — the on.pull_request.paths list and migrate-production's push arm are byte-identical to main.
• Runbook page in apps/docs — deferred to Ship via docs-sync, as the spec specifies.

Reporting the database name (defect found and fixed after the first push)

The first implementation printed - **Database:** ${MONGODB_DATABASE_NAME} to the job summary and was verified only as "the step ran green". It did not satisfy the criterion: GitHub masks every registered secret value wherever it is printed, job summaries included, so the line rendered as `***` and reported nothing. Jamie caught it by reading the summary.

The cause is a classification error, not a formatting one: a database name is not a credential and should never have been a secret. The connection URI is — it carries the password — and stays one. The repo already has precedent: migrate-production reads DEMO_TENANT_CLERK_ORG_ID from vars, and the production database name appears as a literal in the guard's denylist.

Fix: migrate-preview now resolves the name as ${{ vars.MONGODB_DATABASE_NAME || secrets.MONGODB_DATABASE_NAME }}.

• The variable is preferred and is printable, so the summary can name the database.
• The secret remains a fallback, so the job keeps working before the dashboard change and there is no window where preview PRs go red waiting for a human. Deliberate: a lockstep requirement between a merge and a dashboard edit is exactly the failure this run exists to remove.
• Whichever source wins is the value used for the connection, so the reported name cannot disagree with the database actually migrated — the fallback degrades legibility, never honesty.
• DB_NAME_IS_PRINTABLE lets the summary say which case it is in: when the name is still a secret, the summary states that it is masked and how to fix it, rather than showing a bare ***.
• migrate-production is untouched and keeps reading its name from secrets.

Second round: masking is by value, not by source

Adding the environment variable was not sufficient. On re-run (attempt 3) the summary took the printable branch — the "this is masked" clause disappeared, proving vars.MONGODB_DATABASE_NAME resolved — yet the value still rendered as ***. GitHub masks a registered secret's value wherever it appears in a job, regardless of which context produced the string. So while any secret in scope holds that same value, the name cannot be printed by any means: not from vars, not from the migration script's own output.

The remaining holder is almost certainly a repository-level MONGODB_DATABASE_NAME secret: db-audit.yaml reads secrets.MONGODB_DATABASE_NAME with no environment: and connects successfully, so a repo-level secret exists, and the production guard never fired, so its value is not sustentus-prod. That points at the repo-level secret holding the preview database name — which, if true, also means the database audit has been running against preview rather than production, contrary to what the cut assumed.

db-audit.yaml therefore gains the same vars → secrets fallback. It is additive: with only the secret present its behaviour is byte-identical, so this cannot change what the audit connects to. It exists so the repo-level secret can be converted to a variable without leaving that job with an empty database name. This is the one edit in this run outside the spec's touches:, and it is here because the approved "say what was migrated" criterion cannot hold without it.

Resolved. The leftover MONGODB_DATABASE_NAME secret was deleted and the job summary now prints the database name. The sequence that got there, for the runbook docs-sync writes at Ship:

1. Add MONGODB_DATABASE_NAME as an environment variable on preview.
2. Delete every MONGODB_DATABASE_NAME secret in the job's scope — the environment's and the repository's. Step 1 alone is not enough: masking is by value, so one surviving secret holding that string blanks it everywhere, including where it is read from the variable.
3. Re-run the job; the summary names the database.

The runbook must say both steps, in that order, and say why — this is the single least obvious thing about the feature, and the failure mode it produces (***, green check, no error) looks exactly like a bug in the workflow rather than a configuration state.

Notes for Verify

• Finding that changes the spec's premise: the preview secrets already resolve. The spec and stub were written on the belief that the preview environment had no MONGODB_URI / MONGODB_DATABASE_NAME, so the loud-fail would redden every services PR until a human set them. This PR's own migrate-preview run disproves that: both variables printed as *** (GitHub masks a non-empty secret; an unset one prints empty), the production guard did not trip, and the migration connected and reported no pending work. Either the secrets were added between the cut (2026-08-12) and this build, or they resolve from a repo-level fallback that is not named sustentus-prod. The Sequencing risk in the spec is therefore not live — merging this will not redden services PRs.
• What that leaves open is parity, not presence. Nothing in CI can prove the database CI migrates is the database Vercel's preview scope serves — the two values are only comparable by eye, in the two dashboards. That comparison is the remaining half of the first acceptance criterion and the one thing Verify should insist on before the merge tick.
• The single most important thing for Verify to establish: does the preview environment define its own secrets, or is migrate-preview inheriting repo-level ones? The two possibilities are not equally benign, and CI cannot distinguish them from the outside:
  • .github/workflows/db-audit.yaml declares no environment:, so it reads repo-level secrets — and its Audit database job passes, so repo-level MONGODB_URI exists and connects.
  • The cut recorded that repo-level secrets point at production, and the production database is named sustentus-prod. If both of those were true and the preview environment defined nothing, migrate-preview would have inherited the production values and the guard would have aborted the job. It did not abort. So at least one of the three is false.
  • Benign reading: the preview environment defines its own secrets, pointing at the preview DB.
  • Non-benign reading: the environment defines nothing, migrate-preview inherited repo-level values, and the guard did not fire because the production database is not literally named sustentus-prod — in which case the denylist has been decorative all along and a PR may have migrated production.
  • Resolving this is a two-second look at Settings → Environments → preview. If the secrets are listed there, the benign reading holds and nothing further is needed. If that list is empty, stop and treat it as a production-safety question before this merges.
• Separate pre-existing issue, raised and now settled: because db-audit has no environment:, the nightly and per-PR database audit runs against whatever repo-level points at. Deleting the repo-level secret left it with no database name at all, so this run gave it a preflight that refuses to audit blind — and Jamie decided the target: production (sustentus-prod), set as a repository variable. Rationale and the precedence interaction are in 05_verify/output/verify.md → Finding 4.
• Worth a close look: the set -o pipefail in Run migrations (without it the tee masks a failed migration), and the sed redaction pattern covering both mongodb:// and mongodb+srv://.
• inputs.target is empty for pull_request and push events; both if expressions guard it behind an event_name == 'workflow_dispatch' check, so it is never evaluated bare.
• For Ship's docs-sync: two existing pages now describe behaviour that no longer exists and must be corrected alongside the new runbook — apps/docs/app/technical/development/ci-cd ("skip gracefully until a secret is configured") and apps/docs/app/technical/development/database (same claim, plus the migration job description).
• No turbo.json globalEnv change: this run adds no process.env read in repo source. The workflow's env comes from GitHub environment secrets, which are out of turbo's scope.
• No tests added: the diff is CI configuration with no unit-assertable pure logic.

Verify: preview-db-wiring

• production-readiness: run — 2 blockers, 5 advisories. Both blockers were caused by this run's own operator remediation, not by the original diff. Code halves fixed on branch; config halves are the operator's and are listed below.
• code-review: medium (spec complexity: standard) — 2 findings, both real, both fixed on branch. The CI Review diff against CONVENTIONS.md job reported skipped, so the local review ran.
• security-review: not required by the contract's trigger list (no auth, payments, PII or route policies in the diff). The one security-adjacent decision — reclassifying a database name from secret to variable — was covered explicitly by the readiness pass's Secrets & env section and by the two reviews; MONGODB_URI remains a secret everywhere and the summary scrubs connection strings.
• playwright: TODO — manual DoD smoke performed instead.

DoD smoke (each line says who verified it)

The Definition of Done here is not a Vercel preview: this feature's whole surface is the GitHub Actions run and its job summary. The smoke was performed there.

• migrate-preview connects and migrates on a packages/services/** PR, no skip path — demonstrated: run 31698758504, job "Migrate preview database", all 21 migrations up: (agent)
• Job summary names the resolved database and reports the outcome — demonstrated: the summary printed the database name once the leftover secret was deleted (operator)
• The summary contains no connection URI or credential fragment — demonstrated: sed scrub at the tee, plus log inspection of runs 31695629022 / 31698758504 (agent)
• Preview MONGODB_URI + MONGODB_DATABASE_NAME are set, and the name matches the Vercel preview scope — demonstrated: dashboard comparison (operator, reported "verified, all good")
• Missing configuration fails red with an actionable message — verified by inspection of the preflight steps; not exercised live, since exercising it means removing config from a shared environment (agent)
• Production-DB guard still refuses sustentus-prod — logic unchanged; comment rewritten (agent)
• workflow_dispatch targets one database — bare dispatch → preview only; target: production → production only (agent, by inspection of both if: expressions)
• pull_request path filter unchanged, migrate-production push-to-main behaviour unchanged — diffed against main (agent)
• apps/docs runbook — deferred to Ship's docs-sync, per the spec
• auth / payments / notifications: not touched — this diff is CI configuration only, with no route, session, payment or notification code in it (agent)

Findings & cleanup

Blocker 1 — production migrations were running with no database name (fixed on branch, config outstanding)

Verified from the live log on main (run 31699507824, job 94444970108, 12:20 UTC): the production job printed MONGODB_DATABASE_NAME: empty, where a set secret renders as ***. Empty name → connectOptions: { dbName: undefined } → the driver falls back to the URI's path database.

Cause: this run's remediation instructed deleting the repository-level MONGODB_DATABASE_NAME secret so the preview summary could print its own name. migrate-production read secrets.MONGODB_DATABASE_NAME only and had been inheriting that repo-level secret. db-audit was given the vars || secrets fallback before the deletion; migrate-production was not, because the spec placed it out of scope. Out of scope did not mean out of blast radius.

Fixed: same vars || secrets resolution, plus a preflight that fails red on an empty name.

Blocker 2 — production had no reviewer gate, and the claim that it did was load-bearing (config, outstanding)

Same run: the production job was created at 12:19:49 and started at 12:20:02 — no waiting state, so no required reviewers are configured, despite the workflow header and this run's own notes asserting that gate as the reason a target: production dispatch is safe.

One correction to the readiness report, which called this a path added by this PR: it is not. workflow_dispatch already ran migrate-production unconditionally before this branch. Adding the target input made a bare dispatch go to preview, so this change narrows the exposure. The missing reviewer gate is a real and pre-existing hole; it is not a regression from this work, and the documentation asserting it exists is what needs correcting either way.

Finding 3 — a repo-level variable would beat the production environment's secret (fixed on branch)

vars merges repository-level and environment-level variables, and || takes the first non-empty value. So a repository variable named MONGODB_DATABASE_NAME — exactly what this run's remediation advised creating, holding the preview name — would win over the production environment's own secret, and production would migrate under the preview database name with the production URI. The new preflight would not catch it: the name is non-empty.

Fixed: migrate-production now asserts the resolved name equals sustentus-prod. It mirrors the preview job's denylist — preview refuses to be production, production refuses to be anything else.

Finding 4 — db-audit now resolves an empty database name (fixed on branch)

db-audit.yaml declares no environment:, so it resolves repo/org-level values only. It too was inheriting the deleted repo-level secret, so it currently audits whatever the URI path names while reporting green. Fixed: it refuses to run when a URI is present but no name is, rather than auditing blind. That preflight is what turned the Audit database check red on this branch — correctly, and it stays red until the repository variable below exists.

Target decided: production (sustentus-prod) — Jamie's call, asked because it is a product question, not a code one. Leanness is a property of data that actually accumulates, and preview is migrated but barely populated, so production is the only database whose shape is worth tracking. Recorded in the workflow header so the next reader doesn't have to re-derive it.

Two consequences, both already guarded:

• The target is set by a repository variable, not an environment, because this job must run unattended on every PR and on the nightly cron. Giving it environment: production would put the audit behind that environment's required reviewers (blocker 2 wants those added) and stall every PR. Repository-level is the deliberate choice, not an oversight.
• That repository variable is exactly the precedence hazard of finding 3 — repo-level vars merge into every job. It is safe here only because its value is sustentus-prod: migrate-production asserts that name and passes, and migrate-preview has its own environment variable, which outranks the repository one. Were preview's variable ever deleted, preview would resolve sustentus-prod and the denylist guard would abort the job — it fails closed.

The audit is read-only, and db:audit already exits non-zero when it inspects zero collections. So if the repository MONGODB_URI turns out not to be the production cluster, the audit goes red rather than quietly reporting on an empty database — the mismatch is self-detecting.

Advisory 5 — DB_NAME_IS_PRINTABLE proves the variable exists, not that it prints (fixed on branch)

Masking is by value, so a surviving secret holding the same string blanks the name even on the "printable" branch. The hint now appears on both branches instead of only the fallback one.

Advisory 6 — stale docs (deferred to Ship, as specified)

apps/docs/.../ci-cd and .../database still say both jobs "skip gracefully until a secret is configured", which is now false for both, and neither documents the target input. Ship's docs-sync owns these along with the new runbook.

Advisory 7 — the production guard is an unanchored substring match (accepted)

grep -q "$PRODUCTION_DB_NAME" against the URI would also match a host merely containing sustentus-prod. It fails closed, so the direction is safe; noted, not changed.

Operator actions required before the merge gate

1. Settings → Variables → Actions → repository variable: add MONGODB_DATABASE_NAME = sustentus-prod. This is the audit target decided above, and it is what turns the red Audit database check green. It also, by itself, gives migrate-production a correct name.
2. Settings → Environments → production: add MONGODB_DATABASE_NAME = sustentus-prod as an environment variable too. Redundant with (1) today, deliberately: it states production's own name at production's own scope, so changing the repository default later cannot silently re-point production migrations. Until one of (1) or (2) exists, migrate-production fails red — deliberately, because the alternative is what it was doing silently this morning.
3. Settings → Environments → production: add required reviewers, or accept in writing that target: production is a one-click unreviewed production migration for anyone with write access.
4. Answer the incident question: did the deleted repository-level secret hold the preview database name? If so, every merge to main between the demo-tenant work and this fix ran db:migrate up against the preview database while the check read "Migrate production database". Settings → Environments → production answers it; the answer decides config fix vs incident.

Context budget: exceeded the Inputs table deliberately — the readiness pass reads apps/docs and packages/services/scripts, and confirming both blockers required live Actions logs from main rather than the branch diff.

A green migration check now proves a migration actually ran

Who it's for: Admin What shipped: CI now migrates the real preview database, names it in the run summary, and fails red when misconfigured instead of skipping. Why it matters: Schema changes are proven before production — Refine the Bridge / Q2 Objective 3, Validate Technical Infrastructure & Payout Flow.

The same run closed two ways production could have migrated the wrong database.

Dig deeper: <merged-PR URL>

Ship: preview-db-wiring

• pr: #799 · merged: pending — Ready to merge is ticked; the squash follows this commit's checks
• CI: green after the operator set the repository variable. Two checks went red during the run and both were the new guards firing correctly, not defects: Audit database refused to run without a database name, and Migrate production database would have refused an empty one. Both are configuration states this feature exists to make visible.
• technical docs: new preview database runbook; corrections to technical/development/ci-cd and technical/development/database, which both still said the jobs "skip gracefully until a secret is configured" — false for both since this change
• business docs: no business docs impact — CI configuration, no user-visible behaviour, no persona capability or journey step altered
• release notes: ship-note only — infra/reliability, nothing an end user can see or do differently, so no changelog entry (Ship step 3, infra branch)
• sent: ship note queued for #product-update by the merge

Acceptance check (vs spec)

• Preview environment has MONGODB_URI + MONGODB_DATABASE_NAME, matching the Vercel preview scope — operator-verified across both dashboards ("the criteria have been verified, it's all good"); the name is now a variable, so the summary prints it and the comparison is a glance
• migrate-preview connects and migrates on a packages/services/** PR, no skip path — run 31698758504, all 21 migrations reported up:
• The summary names the database and reports the outcome, with no URI — confirmed live once the leftover secret was deleted; connection strings are scrubbed at the tee
• Missing configuration fails red with an actionable message — demonstrated twice in this run, unintentionally and correctly, on db-audit and migrate-production
• The production-DB guard still refuses sustentus-prod — logic unchanged; comment rewritten
• workflow_dispatch targets one database — bare dispatch → preview; target: production → production
• pull_request path filter and migrate-production's push-to-main behaviour unchanged
• Preview-database runbook exists in apps/docs — secret/variable location, the Vercel-parity requirement, rotation, and catch-up dispatch, plus the masking trap that cost this run two rounds

Carried forward (not blocking this merge, still open)

1. production has no required reviewers. Verified from the live run on main: the job was created at 12:19:49 and started at 12:20:02, with no waiting state. The workflow header and the docs both asserted that gate; this PR's docs no longer claim it, and the target: production dispatch remains a one-click production migration for anyone with write access until reviewers are added. Pre-existing — workflow_dispatch ran migrate-production unconditionally before this branch, so the target input narrowed the exposure rather than creating it.
2. The incident question is unanswered. Did the deleted repository-level MONGODB_DATABASE_NAME secret hold the preview database name? If it did, merges to main between the demo-tenant work and this fix ran db:migrate up against preview while the check read "Migrate production database". Settings → Environments → production answers it. Worth answering even now that the configuration is correct.

Context budget: within the Inputs table — the two docs pages, the spec, and the verify record.