Skip to Content

← All archived runs

Run: simplified-status-workflows

run.md

Run: simplified-status-workflows

  • branch: claude/wonderful-johnson-hog8cg
  • pr: #486

01_define/output/spec.md

Spec: Simplified status workflows (deterministic JSON)

  • slug: simplified-status-workflows
  • personas: admin, csm, sdm, expert, vendor, customer
  • touches: packages/services/src/db/{workflows (new JSON + Zod loader),models (status field types → string on lead/quote/proposal/invoice + status-history),services/status,services/leads,services/proposal,services/quote,services/invoice,seed}, a one-shot migration/cleanup script (packages/services migrations), apps/web/app/(app)/admin/status, apps/web/lib/admin-status-schema.ts, apps/dashboards (status display), apps/docs/app/business/service-journey (reconciled in Release)
  • complexity: complex

Problem

The status system is a fully-configurable, DB-backed graph engine. Each status is a row in a status collection scoped to a workflow enum (lead | invoice | quote | proposal | milestone | verification), carrying an embedded transitions[] array (toStatus ObjectId, type of positive | negative | loop, optional condition). Display order is recomputed at runtime by BFS graph traversal (workflow-display-order.ts), the journey UI is assembled by merging statuses with history (build-progress-items.ts), and the whole thing is editable through admin CRUD at /admin/status/*. The trigger logic is scattered across the service layer (leads/lead-status-progress.ts, proposal/manager-review-proposal-flow.ts, quote/customer-quote-response.ts, invoice/index.ts, milestones/.../status/route.ts), each resolving a status by name and writing a StatusHistory row.

This flexibility is unused: the workflow has been determined and will change only minimally. The configurable engine has cost rather than benefit — it has drifted from the documented workflow (the live admin set contains lead statuses like quotation_process, awaiting_confirmation, survey_sent, work_in_progress, completed that the service-journey docs and reference-data.ts seed don't describe, and the live invoice/quote transition graphs look incomplete), it requires runtime graph traversal to derive ordering that is in practice fixed, and it has no way to express the two things the product actually needs: workflows nested inside workflows (the lead journey contains the proposal/quote/invoice/milestone sub-journeys) and per-persona display names (the same stage reads differently to a customer vs an internal user).

Reliable, correct status progression underpins the entire service journey end-to-end — the bridge that Build the Bridge exists to deliver and Refine the Bridge exists to harden. It directly serves Q2 2026 Objective 3 (Validate Technical Infrastructure & Payout Flow): lead transactions completed end-to-end with zero failures and a tight lead-to-payout cycle depend on a deterministic, non-drifting workflow rather than an editable graph that can be misconfigured into an invalid state.

Proposed change

Make a single, versioned, in-repo JSON file the source of truth for the workflow: every status, its transitions, its colour, its default display name and per-persona label overrides, organised as a nested tree where the sub-workflows sit inside the lead stage they belong to. The configurable DB write-path is retired; the admin screen becomes a read-only viewer of the JSON; runtime trigger logic validates against the JSON and hard-rejects any transition the JSON does not permit. Per-entity audit data (StatusHistory) stays in MongoDB unchanged.

1. The workflow JSON (canonical)

  • A single JSON document (e.g. packages/services/src/db/workflows/workflows.json) plus a Zod schema
    • typed loader in @sustentus/services that validates it once at load and exposes typed accessors.
  • Each status node carries: name (stable key), displayName (default), labels (optional map of persona → label for admin | csm | sdm | expert | vendor | customer), hex, description?, tag?, isActive, and transitions[] where each transition targets another status by name (not ObjectId) with a type of positive | negative | loop and optional condition.
  • The tree is nested: top-level is the lead journey; the proposal, quote, invoice and milestone sub-workflows are embedded inside the lead stage during which they run. The JSON also makes ordering explicit (declared order in the file), so no BFS traversal is needed to render the journey.
  • verification is not part of the lead journey. Expert evidence proves an expert is an expert — it is attached to the expert profile, not to a lead or milestone. So the verification workflow is not nested anywhere in the lead tree; it stands as its own top-level, expert-profile-scoped workflow in the JSON (statuses draft → submitted → approved | rejected, with rejected → draft loop). The milestone sub-workflow carries no verification step.

Proposed workflow tree — RED-LINE THIS

Reconstructed from the service-journey docs, the trigger code, and the admin screenshots (screenshot nomenclature is canonical per the agreed decision). Names/labels/colours are taken from the screenshots; the edges and the nesting are my best reconstruction — please correct any wrong edge, ordering, label, or nesting here, and that steers Build. Where the live config looks incomplete (quote and invoice transition counts), I have proposed the documented happy path.

lead  (the parent journey)
├─ pending              "Pending"               #000000   → backlog (positive), qualified_out (negative)
├─ backlog              "Backlog"               #bdbdbd   → quotation_process (positive), qualified_out (negative)
├─ quotation_process    "Bid Process Started"   #8884d9   → awaiting_confirmation (positive)
│   └── proposal  (sub-workflow: experts bid)
│       ├─ proposal_submitted   "Submitted"   → proposal_accepted (positive), proposal_rejected (negative)
│       ├─ proposal_accepted    "Accepted"    (terminal)
│       └─ proposal_rejected    "Rejected"    (terminal)
├─ awaiting_confirmation "Quote Review"         #8884d9   → work_in_progress (positive), quotation_process (loop, on reject)
│   └── quote  (sub-workflow: customer decides on quote)
│       ├─ quote_draft       "Draft"     #000000  → quote_submitted (positive)
│       ├─ quote_submitted   "Submitted" #80ffff  → quote_accepted (positive), quote_rejected (negative)
│       ├─ quote_accepted    "Accepted"  #000000  (terminal)
│       └─ quote_rejected    "Rejected"  #000000  (terminal)
├─ work_in_progress     "Work In Progress"      #dcff2e   → delivered (positive)
│   ├── milestone  (sub-workflow: delivery, per milestone)
│   │   ├─ upcoming      → in-progress (positive)
│   │   ├─ in-progress   → done (positive), blocked (negative)
│   │   ├─ blocked       → in-progress (loop)   ← loops back into the workflow until unblocked
│   │   └─ done          (terminal; triggers an invoice)
│   └── invoice  (sub-workflow: per completed milestone)
│       ├─ invoice_draft     "Awaiting approval" #64748b → invoice_submitted (positive)
│       ├─ invoice_submitted "Submitted"         #000000 → invoice_approved (positive), invoice_rejected (negative), invoice_disputed (negative)
│       ├─ invoice_approved  "approved"          #000000 → invoice_sent (positive)
│       ├─ invoice_sent      "Awaiting payment"  #ca8a04 → invoice_paid (positive)
│       ├─ invoice_disputed  "Disputed"          #f97316 → invoice_submitted (loop)
│       ├─ invoice_paid      "Paid"              #ffffff (terminal)
│       └─ invoice_rejected  "Rejected"          #000000 (terminal)
├─ delivered            "Delivered"             #78ff75   → survey_sent (positive)
├─ survey_sent          "Survey Sent"           #8884d9   → completed (positive)
├─ completed            "Completed"             #8884d9   (terminal)
└─ qualified_out        "Qualified Out"         #bca9a9   (terminal, negative)

Per-persona labels are not in the screenshots (display names are global today). I will leave labels empty in the seed JSON and the default displayName applies to everyone, except where you tell me a persona should differ. Add the overrides you want during spec review (e.g. customer sees "We're reviewing your quote" while internal users see "Quote Review") and Build encodes them.

2. Read path → JSON

  • A typed workflow/status accessor reads from the JSON (replacing the runtime graph helpers getByWorkflow, getDisplayOrder, getInitialStatusForWorkflow, BFS ordering).
  • Display resolution takes the viewer's persona and returns the persona override if present, else the default displayName.
  • build-progress-items.ts keeps merging StatusHistory (per-entity, from the DB) with the now-JSON status catalogue to render the journey strips.

3. Migration — convert to string keys and clean up stale data

Entities (lead, quote, proposal, invoice) and StatusHistory currently reference a status by ObjectId into the status collection (lead.status: { ref: "status" }, etc.). Rather than preserve those refs, a one-shot migration script converts the system to the JSON model and removes the stale data outright:

  1. For each entity and each StatusHistory row, resolve its current status ObjectId to the status's name (via the status collection as it exists pre-migration) and rewrite the field to that string key (the stable JSON name). Rows whose status no longer exists in the JSON (drift / removed statuses) are remapped to the closest valid status or flagged, per a small mapping table in the script — no silent data loss.
  2. The model field types change from ObjectId ref to string (status: string, validated against the JSON's known names).
  3. Drop the status collection and any orphaned/stale status records once entities and history no longer reference it. The JSON is the only source thereafter.

We are explicitly not backfilling or maintaining a projected collection — the script is a clean cutover. It is idempotent and safe to re-run.

4. Admin screen → read-only viewer

  • /admin/status (list + detail) stays, rendering directly from the JSON. Colour, default display name, per-persona labels, and transitions are all read-only.
  • The create/edit/delete pages and their server actions (createStatus, updateStatus, deleteStatus) and the statusService write methods (create, update, softDelete) are removed. apps/web/lib/admin-status-schema.ts's write/parse schemas go with them.
  • Access stays gated to admin | csm | sdm (now read-only).

5. Trigger validation → hard-reject

  • The scattered trigger points keep their entry signatures but route their transition decision through one JSON-backed validator: given (workflow, fromStatusName, type) (or an explicit target), it returns the permitted target or throws if the JSON does not permit it. Invalid transitions can no longer occur — the JSON is a real guardrail.
  • StatusHistory writes are unchanged (still one row per applied transition).

Acceptance criteria

  • A single versioned JSON file in the repo defines every status for all six workflows, with name, default displayName, optional per-persona labels, hex, transitions (targeting by name, typed positive | negative | loop), and is validated by a Zod schema at load.
  • The JSON is structured as a nested tree — the proposal, quote, invoice and milestone sub-workflows are nested inside the lead stage during which they run. verification is a separate top-level, expert-profile-scoped workflow and is not nested in the lead tree.
  • The milestone sub-workflow includes a blocked status that loops back into the workflow (in-progress → blocked → in-progress) and carries no verification/evidence step.
  • Status display resolves by the viewer's persona: a persona override is shown when present, otherwise the default display name. (Demonstrable for at least one status with an override.)
  • The status catalogue (names, display names, colours, ordering, transitions) shown anywhere in the product matches the JSON exactly and the live admin screenshots' nomenclature — no value is read from a hand-editable source.
  • /admin/status renders as a read-only viewer: no create, edit, or delete is possible (pages/actions removed); the list and detail still display correctly for admin | csm | sdm.
  • A trigger attempting a transition the JSON does not permit is hard-rejected (throws / errors, status unchanged); a permitted transition succeeds and writes a StatusHistory row as before.
  • A migration script converts entity and StatusHistory status fields from ObjectId refs to string name keys, drops the status collection and clears orphaned/stale status data; the model field types are now string. After it runs, leads/quotes/proposals/invoices and their history resolve correctly against the JSON with no broken references in the journey UI.
  • Lead progression, proposal accept/reject, quote accept/reject, invoice approve/send/dispute/ reject/pay, and milestone advance all continue to work end-to-end against the JSON.

Out of scope

  • Preserving the existing status collection or backfilling a projected copy — the migration drops it and cleans up stale data instead.
  • Adding any new statuses, transitions, or workflows beyond capturing the current determined set (the milestone blocked status is in scope).
  • Building an admin UI to edit the JSON (it is edited in-repo via PR; the screen is read-only).
  • Reconciling the apps/docs service-journey pages and the reference-data.ts seed to the new nomenclature — this rides in the Release stage via docs-sync, not Build.
  • Per-persona theming/colour (only labels vary by persona; hex is global).
  • Notification, SLA, or activity-log changes keyed off statuses (unchanged).

Open questions

  • Non-blocking: exact per-persona label overrides are empty in the seed JSON pending your input during spec review; the default display name applies until you specify overrides. This does not block implementation — Build ships the override mechanism regardless and the seed values are data edited in the spec/JSON.
  • Non-blocking: a few live edges (quote and invoice transition counts in the screenshots look incomplete) are filled with the documented happy path in the proposed tree above; correct them in review. The mechanism is unaffected by which exact edges are chosen.

02_build/output/notes.md

Build notes: simplified-status-workflows

  • commits: workflow JSON + Zod schema + loader; JSON-backed StatusService; model field-type changes (status → string) + StatusHistory; build-progress/display rekeyed by name; one-shot migration; seed trimmed; admin read-only viewer; service triggers rewired to string keys + assertTransition; typecheck fixes.

What changed

New — packages/services/src/db/workflows/

  • workflows.json — the single source of truth: a nested tree with lead as the parent journey; proposal, quote, invoice, milestone nested inside the lead stage they run in; verification as a separate top-level (expert-profile-scoped) workflow, not nested. Each status carries name, displayName, optional per-persona labels, hex, isActive, and transitions (target by name, typed positive | negative | loop). The milestone sub-workflow includes blocked with a loop back to in-progress. Invoice edges were aligned to the actual code flow (invoice_draft → sent/rejected/disputed, sent → paid) so the guardrail never rejects a real transition.
  • schema.ts — Zod schema (recursive) validating the file shape.
  • index.ts — loads + validates the JSON once at import, flattens the tree into statusByName / byWorkflow maps, validates every transition target exists in the same workflow, and exposes accessors + the guardrail (getStatus, getStatusesByWorkflow, getWorkflowTree, resolveStatusView, resolveStatusLabel, transitionTargetByType, isTransitionAllowed, assertTransition).

Source of truth + reads

  • StatusService rewritten to be JSON-backed (sync): findByName, findAll, getByWorkflow, getDisplayOrder, tree, resolveView, resolveLabel, nextStatus, isTransitionAllowed, assertTransition. Removed create/update/softDelete/findById/ findIdByName/ensureInvoiceWorkflowStatuses.
  • models/status.ts no longer defines a collection — re-exports IStatus/ITransition as the JSON-backed shapes. workflow-display-order.ts + build-progress-items.ts rekeyed from ObjectId to status name.
  • StatusView ({name, displayName, hex}) added to models/types.ts; service read methods map the persisted status string → a StatusView so existing .displayName/ .hex consumers keep working.

Models → string status keys

  • lead, quote, proposal, invoice and StatusHistory status fields changed from ObjectId ref to string. *Populated.status typed as StatusView.

Migration

  • migrations/1781913600000-status-to-string-keys.ts — converts every entity + StatusHistory status ObjectId → the status name string (via the legacy statuses collection), clears orphaned/stale refs, then drops the statuses collection. One-way cutover (no backfill); down is a documented no-op.
  • seed/reference-data.ts no longer seeds statuses (JSON is canonical).

Triggers → string keys + hard-reject guardrail

  • Lead progression, proposal accept/reject, quote accept/reject, invoice approve/send/dispute/reject/pay, milestone advance, lead qualify-out, BRD/backlog promotion — all set the string name directly and call assertTransition where a current status exists. .populate("status") removed everywhere; reads resolve to a StatusView.

Admin → read-only viewer (apps/web)

  • Deleted create/edit pages, actions.ts, status-form, status-delete-button, status-detail-actions, and lib/admin-status-schema.ts. /admin/status list + detail render directly from the JSON (keyed by name, transitions show t.to, optional per-persona labels). Removed the "Create status" button.

Acceptance criteria status

  • Single versioned JSON defines all six workflows, Zod-validated at load.
  • Nested tree; verification separate top-level, not nested in the lead tree.
  • Milestone blocked status loops (in-progress → blocked → in-progress); no verification step.
  • Status display resolves by viewer persona (override → default).
  • Catalogue matches the JSON everywhere; nothing read from a hand-editable source.
  • /admin/status is read-only (create/edit/delete removed); list/detail still render.
  • Illegal transitions are hard-rejected (assertTransition throws); valid ones write StatusHistory.
  • Migration converts entity + history status to string keys and drops the statuses collection.
  • Lead/proposal/quote/invoice/milestone flows compile and build against the JSON end-to-end.

Verify result

  • Format · Lint · Typecheck · Audit database — all green in CI on the head commit.
  • Vercel preview build (web + marketing) — READY (real next build).
  • Local pnpm typecheck: services + dashboards + all packages pass; the only web diagnostic was a stale local .next/types validator referencing the two deleted admin routes — absent on a fresh CI/Vercel build (confirmed green there).

Notes for review

  • Migration runs on merge (db-migrate workflow) and is irreversible by design — it drops the statuses collection. Review the orphan-handling (stale refs are cleared).
  • Reconstructed edges: the workflow graph (esp. quote/invoice) was reconstructed and aligned to real code flows; per the spec these edges were non-blocking — sanity-check the JSON against intended product behaviour.
  • Per-persona labels ship empty (default display names apply) pending product input; the override mechanism is in place.
  • Docs/service-journey reconciliation to the new nomenclature is deferred to Release (docs-sync), per the spec.

03_release/output/investor-update.md

Status workflow is now deterministic and tamper-proof

Who it's for: Internal teams (admin, CSM, SDM) and every customer journey that moves through a status What shipped: The lead-to-payout status workflow moved to a single version-controlled definition that hard-rejects any invalid transition. Why it matters: Removes a class of misconfiguration risk from every transaction — advancing Refine the Bridge and Q2 objective "Validate technical infrastructure & payout flow".

Invalid status changes are now impossible by construction and verified in CI.

Dig deeper: https://github.com/sustentus/sustentus/pull/486

03_release/output/release.md

Release: simplified-status-workflows

  • pr: https://github.com/sustentus/sustentus/pull/486 · merged: pending (squash, on ticked "Ready to merge")
  • CI: green on the build head (Format · Lint · Typecheck · Audit · Vercel previews); re-verified after the docs + review-fix commits
  • technical docs: updated technical/development/database (dropped statuses collection references; workflow now in JSON)
  • business docs: updated business/service-journey/{page,proposals,quote,invoicing,delivery,lead-intake}, business/roles (admin → read-only status viewer), business/platform-overview (status workflow no longer per-tenant configurable)
  • release notes: investor-only (internal/infra change) — investor draft in 03_release/output/investor-update.md; no end-user changelog (no customer-facing behaviour change)
  • deploy: production deploy verified via Vercel MCP after merge (touched apps + help)
  • sent: none — investor email withheld at user's explicit request ("don't send the investor email")

Review summary

  • [bug] milestone/index.ts findByProposalOrderedWithInvoiceSummary kept a nested .populate("status") on the invoice — invoice status is now a string, so strictPopulate would throw at query time on the expert's accepted-lead view. Fixed: dropped the nested populate, resolve the string via statusService.resolveView.
  • [risk] manager-review-proposal-flow.ts could call assertTransition("quote_submitted","quote_submitted") (no self-loop in JSON → throw) on an already-submitted quote. Fixed: promote only from quote_draft; already-submitted is a logged no-op.
  • [nit] Migration clears un-remappable statushistories.status (leaving a required field unset on historical rows) — accepted; one-way cutover, validation only runs on save.
  • Clean: lead/quote/proposal/invoice triggers use string status keys + assertTransition correctly; every code transition exists in workflows.json; StatusView mapping consistent; no leftover findIdByName/getStatusModel/.populate("status")/status._id for statuses.

Acceptance check (vs spec)

  • Single versioned JSON for all six workflows, Zod-validated at load — workflows.json + schema.ts + loader.
  • Nested tree; verification separate top-level, not nested in the lead tree.
  • Milestone blocked status loops (in-progress → blocked → in-progress); no verification step.
  • Per-persona display resolution (override → default) via resolveStatusLabel/resolveView.
  • Catalogue matches the JSON everywhere; nothing read from a hand-editable source.
  • /admin/status is read-only (create/edit/delete removed); list/detail render from JSON.
  • Illegal transitions hard-rejected (assertTransition throws); valid ones write StatusHistory.
  • Migration converts entity + history status → string keys, drops the statuses collection, clears orphans.
  • Lead/proposal/quote/invoice/milestone flows work against the JSON (CI build green; review-confirmed edges).