Skip to Content
TechnicalApplicationsTenant management console

Tenant management console

The console is the Sustentus-side operational surface: the app the Sustentus team and its partners use to administer tenants, separate from the tenant-facing platform in apps/web.

Overview

Package name: @sustentus/console Framework: Next.js 16 (App Router, Turbopack) Port: 3006 Path: apps/console/ Vercel project: tenant-management

Who it is for

The console has two audiences, and they are not the platform’s six roles. A platform role (vendor, customer, expert, CSM, SDM, admin) is scoped to a tenant and governed by the permissions registry. A console audience is scoped to Sustentus:

AudienceWhoSees
sustentusThe Sustentus teamEvery live tenant; can invite and revoke access
partnerA vendor partnerOnly the tenants assigned to them

Access model

Console users are org-less Clerk users carrying a publicMetadata marker:

{ "consoleAccess": { "audience": "sustentus" } }

parseConsoleAccess (in @sustentus/services/shared) is the one reader of that shape. Three properties follow from it:

  • Deny by default. proxy.ts gates every route the app will ever have, not a chosen segment. A signed-in user with no marker reaches /no-access and nothing else.

  • The marker is read live from Clerk on every request, not from a session claim — so clearing it denies the very next request rather than waiting for a session to expire. The Clerk read behind readViewer is wrapped in React cache, so it is one call per request however many components ask.

  • The proxy gates console access, not every staff-only capability. Its staff-matched routes are /access(.*), /tenants/deleted(.*), /notifications/delivery(.*), /notifications/fleet(.*), /prompts(.*), /assistant-health(.*), /agent-runs(.*), /assistant-switch(.*) and /api/prompts(.*); the rest are open to both audiences, because partners legitimately read the tenants assigned to them. A staff-only action on such a route carries its own requireStaff(), and that call is the whole boundary — server actions are dispatched by action id and can be posted from any path, so route matching never constrains them.

  • Staff answer the email-code gate first. After the marker, proxy.ts asks decideConsoleGate (lib/console-gate.ts) whether a Sustentus staff member holds a gate pass issued to their own primary Clerk email; without one, every page is rewritten to the code screen at /gate (the address bar keeps the page asked for, which loads once the code is answered) and every /api/** path answers a bare 404. Partners never meet it. It is the demo hosts’ gate — the same roster, secret and screen (@sustentus/services/shared → access-gate), so one answered code opens the console, demo and explore (Demo environment → One code opens console, demo and explore). The console mails a code only to the signed-in staff member’s own address, and logs a warning naming their Clerk user id when that address is not on the roster. With DEMO_ACCESS_ROSTER or DEMO_ACCESS_SECRET unset or malformed — a preview, say — staff keep the Clerk login alone and the proxy logs a warning once per instance; production needs both, with the platform’s secret value, plus DEMO_HOSTS. The proxy is not the only line: the viewer checks every page, route handler and server action calls (requireViewer, requireStaff, readStaffViewer in lib/console-access.ts) take the same decideConsoleGate decision on the same inputs, so a staff member without their own pass is sent to /gate by a page or action and refused by a route handler even on a path the proxy did not run on. Only the code screen and its two routes (/gate, /api/gate, /api/gate/request) read the viewer before the gate, through requireViewerBeforeGate and readStaffViewerBeforeGate.

The only routes reachable without a marker are /sign-in, /accept, /api/health and /api/access-gate/hop — the last only sets a pass carried in from a demo host, and whether that pass opens the console is decided by the gate above. /accept redeems a Clerk invitation ticket and is the app’s sole sign-up surface; it refuses anyone without a ticket. /api/health is the probe health-check.sh reads after every merge — matched exactly, it answers only a status word from a MongoDB ping (200 or 503).

Routes

RouteAudienceWhat it does
/BothDashboard: tenant counts and recent activity
/activityBothThe activity trail, newest first, scoped to the viewer
/notificationsBothThe notification centre — the notifiable trail entries in scope, with read state
/notifications/deliverysustentusRecent email delivery failures, newest first
/notifications/fleetsustentusPlatform notification delivery failures across every tenant, newest first, filterable by tenant, notification type and channel
/assistant-healthsustentusStore calls the agentic assistant could not make, newest first
/tenantsBothSearchable, server-paginated tenant directory, filterable by go-live readiness and setup state
/tenants/newBothCreate a tenant and invite its first administrator
/tenants/:idBothTenant detail, its users, its subscription, its go-live readiness, and — staff only — its configuration (profile, registered settings, the six per-stage SLA definitions, and the Roles and access panel for role templates and per-user permission overrides, and the Catalogue panel for its services, products, skills and industries), its partner and lifecycle controls
/tenants/deletedsustentusDeleted tenants, with the recovery window remaining on each
/tenants/deleted/:id/removesustentusThe typed-name confirmation for permanent removal
/ai-spendBothMetered AI spend per tenant for a chosen month, with the population, period and basis stated beside it, and recent generation ids for a manual reconciliation against the gateway
/accesssustentusInvite and revoke console access
/promptssustentusThe prompt workspace as its ICM tree, plus the code-owned rails and routing read-only
/prompts/registers/:keysustentusOne register over time: its publish timeline, the run quality either side of each publish, its error and refusal rates by month, and its scored evals
/prompts/registers/:key/testsustentusA real, streamed conversation on one register’s prompt — every saved draft applied, or what is live — bound to the demo tenant and saved nowhere
/api/prompts/testsustentusPOST — the route the test conversation streams from; gated by the proxy and by its own staff read, answering 403
/prompts/*sustentusOne workspace file: edit or upload it, diff it, preview what it assembles to, publish it, roll it back
/agent-runssustentusWhat the assistant’s turns actually did — failures first, with the register, the model, the outcome, the tool calls and the prompt version set behind each
/assistant-switchsustentusThe platform-wide assistant kill switch, and the tenants currently switched off on their own
/no-access—Where a signed-in user with no marker lands

Tenant directory

/tenants lists live tenants by name, 25 to a page, with a case-insensitive substring search and two filters — go-live readiness and setup state (see Fleet readiness below). /tenants/:id shows the tenant’s identity fields, derived status and setup state, its go-live readiness, and its user list read from Clerk organization memberships (capped at 100 — the page says so when the cap is hit, rather than reading as complete). That list is operable: see Tenant user management below.

Scoping is applied inside the query, never after the read. The rules are pure functions in packages/services/src/db/services/tenant/console-scope.ts:

buildConsoleTenantListFilter(scope, search?, onboardingStatus?) // → the list filter for this audience buildConsoleTenantIdFilter(scope, tenantId) // → the detail filter, or null if malformed

A partner’s filter carries their partnerId; a staff filter carries no partner clause; both exclude soft-deleted tenants. Because the bound is in the filter, a tenant that is not assigned to the requesting partner, a tenant id that does not exist, and a malformed id are indistinguishable to the caller — all three are the same not-found. Nothing on a partner’s directory reveals that other tenants exist: no global count, no pagination past their own set.

partnerId is declared on the Tenant schema and indexed by migration, and written from the tenant detail view — see Partner assignment below.

Fleet readiness

The six go-live checks — minimum catalogue, SLA, service model, integrations, roles, commercial — are scored across the fleet, not re-derived here. The computation is computeTenantReadiness in @sustentus/services/shared, the same function the tenant’s own admin dashboard calls, so the console cannot come to disagree with the dashboard that tenant’s admin is reading.

The directory’s Go-live column names each tenant’s failing checks rather than counting them — the point of a fleet view is to say what to chase. apps/console/lib/console-readiness-gaps.ts maps each unmet check to where it is fixed: staff get an in-page anchor to the panel on tenant detail that clears it (Catalogue, SLA, Roles and access) — the integrations and pricing gaps name the tenant-side page instead, since no console panel connects an integration or sets a list price — and a partner — who sees the same readiness but holds none of those panels — gets the tenant-side page named instead. The catalogue gap resolves from the gap’s first shortfall: staff land on that section of the Catalogue panel (Products, Services or Skills — each section has its own anchor, from lib/console-catalogue-anchors.ts), a partner is told the matching /admin/settings/* page. The Skills section writes each skill with the services it is linked to (at least one) through skillService, so a skills-only shortfall is cleared from the console. Integrations have no console panel, so that gap names /admin/integrations for both audiences.

Only one of the two filters is a query. Setup state is a stored field (tenant.onboarding.status), so it goes into buildConsoleTenantListFilter and the pager’s total stays the size of the filtered list; a tenant whose onboarding subdocument is absent reads as not_started there, exactly as it does everywhere else. Readiness is derived from four other collections and cannot be sorted or skipped on, so the readiness filter reads the whole scoped list (bounded by CONSOLE_TENANT_SCAN_LIMIT, 500), scores it in the same four reads, filters, and pages in memory — the cost is the wider read, not more of them. The page says so when that bound is hit, rather than truncating silently.

The scope rides in the service call on both paths, so a partner’s list is bounded before readiness is computed: the filtered directory can no more reach an unassigned tenant than the unfiltered one can. And readiness never blocks either page — it is derived from collections these views did not previously depend on, so a failure leaves the tenants themselves rendered and says the readiness is unavailable.

Tenant lifecycle

A tenant is created, suspended, ended and — eventually — erased from this app. The states live in an optional lifecycle subdocument on the Tenant model:

{ status: "active" | "suspended" | "deleted", changedAt, changedBy, deletedAt? }

Absence reads as active. Every tenant that predates the feature has no lifecycle field and is active by that fact, so the migration backfills a stamp for the record rather than to make the reads work.

Who holds what

Creating is the one lifecycle action both audiences hold. /tenants/new takes a name and a first-administrator email, creates the Clerk organisation, then the Tenant document, then sends the invitation — Clerk first, so a failure part-way leaves no Tenant document whose organisation does not exist. A partner’s tenant is assigned to them in the same flow, through the same tenantService.setPartner staff assignment uses; without that it would be a tenant they cannot see.

Suspend, delete, recover and permanent removal are requireStaff(), without exception — a partner administers their clients, they do not decide whether a client still has a workspace. The route gate above covers /tenants/deleted(.*), but /tenants/:id is deliberately open to both audiences, so the suspend and delete actions carry their own check and that check is the boundary.

What each state means

  • Suspended — the tenant stays in the directory for both audiences, badged. Its people cannot reach the platform (see Enforcement in apps/web). Reactivating is one click and restores nothing, because nothing was removed.
  • Deleted — soft. The tenant leaves both directories, its people are turned away, and it appears on the staff-only /tenants/deleted list with the days left in a 30-day recovery window. Nothing has been destroyed: recovery returns it to active with its organisation, members and data intact. Past 30 days the page offers no Recover control, but the tenant is still there — nothing removes a tenant unattended.
  • Permanently removed — irreversible. The Clerk organisation is deleted, every tenant-scoped collection is emptied of that tenant’s rows — the activity trail among them — and the Tenant document is dropped. One trail entry survives, tenant-less: see Immutability below.

lifecycle.status: "deleted" and the schema’s isDeleted flag are written and cleared together, by one function — buildLifecycleUpdate in packages/services/src/db/services/tenant/lifecycle.ts. They cannot disagree because nothing else writes either of them, and a unit test pins that.

Reading a deleted tenant

softDeletePlugin hooks pre(/^find/) and ANDs a live-only clause onto every model query, so a deleted tenant is invisible to Tenant.find — which is the plugin working, not a bug to route around. The deleted list, the recovery read and the removal therefore go through the raw driver, under the plugin. Two consequences worth knowing:

  • Those reads pass the schema’s { locale: "en", strength: 2 } collation explicitly. The driver handle knows nothing of the schema, and a string comparison issued under the collection’s default collation cannot use a collated index — clerkOrgId_1 among them.
  • buildConsoleDeletedTenantFilter takes no ConsoleTenantScope argument at all. There is no partner shape to construct, so a partner-visible deleted read is not something a call site can build wrong.

Permanent removal

The confirmation is the tenant’s name, typed exactly, and it is checked server-side against the record — not against a hidden field, which would let anyone posting the action supply both halves of their own check.

The order is the Clerk organisation first, then the data and the Tenant document together. Identity leads because it is the half that can fight back: while the organisation lives, a member signing in still carries its orgId and would have upsertFromClerk write a fresh Tenant document over the purge in progress. The cost is that an interruption leaves a Tenant document with no organisation — which recovery detects and refuses, rather than handing back a workspace nobody can sign in to.

Erasure is policy-driven, not a hand-kept list. tenant-purge was extracted from the demo-reset service so that both empty every tenant-scoped collection from the same lists the existing drift test already guards. A collection added to the platform is covered by both, or by neither.

upsertFromClerk refuses a deleted tenant rather than resurrecting one. The plugin hides the row from that method’s own findOneAndUpdate, so without the guard the first sign-in by one of its users would silently insert a second Tenant document for the same clerkOrgId.

Enforcement in apps/web

The platform’s proxy redirects any authenticated request whose tenant is suspended or deleted to a public /workspace-unavailable page. It runs before the route policy, because the question is whether the workspace may be used at all — not which of its pages this role may see. The page names no cause: a suspension is a commercial or administrative matter between Sustentus and the account, and a customer-facing page is not where that gets explained.

The gate fails open on a read error, logged. It is the one read here that cannot be narrowed to a role or a path, so letting it throw would turn a database blip into an error on every page of the platform at once — to defend a state that is administrative rather than a security boundary, and with the database down there is nothing behind the gate to reach anyway.

Partner assignment

/tenants/:id carries an Assigned partner panel, rendered and operable for sustentus only. Staff set, change, or clear the tenant’s one partner from it — the write that makes a partner’s scoped directory non-empty.

  • The assignment is a field, not a membership: the optional partnerId, holding the assigned partner’s Clerk user id. One partner per tenant is the field’s cardinality, not a rule that has to be policed. Clearing $unsets it, so an unassigned tenant reads the same whether it was never assigned or was cleared.
  • One shared write. tenantService.setPartner(tenantId, partnerId | null) resolves the tenant through buildConsoleTenantIdFilter({ audience: "sustentus" }, tenantId), inheriting the live-only guard and the malformed-id answer. It performs no authorization of its own — deliberately, so that a partner-created tenant can later be auto-assigned to its creator through the same function. Authorization belongs to the caller.
  • The partner is chosen two ways: a select of known partners, and an assign-by-email fallback for someone granted access directly onto an existing account, who leaves no invitation behind and so never appears in the roster. Both resolve to a Clerk user whose marker is re-read live and must still say audience: "partner" before anything is written.
  • The roster has one implementation — readConsoleRoster in apps/console/lib/console-roster.ts, shared with /access. Clerk’s user API cannot filter by publicMetadata, so there is no “list every partner” query; both surfaces derive the same answer from the console’s invitation list.

Tenant user management

From /tenants/:id, both audiences can invite a user, revoke a pending invitation, remove a member, and change a member’s role — Sustentus staff on any tenant, a partner on the tenants assigned to them. The server actions live in apps/console/app/(console)/tenants/[tenantId]/actions.ts.

Every operation writes to the tenant’s Clerk organisation, which is authoritative for who belongs and in which role — the same record the roster reads. The platform’s existing organizationMembership.created|updated|deleted webhook mirrors each change onto the Mongo user collection; the console adds no sync path and writes no user document itself. A change made here is what the tenant’s own platform sees on its next request.

The assignable roles are the platform’s. CONSOLE_ASSIGNABLE_ROLES in apps/console/lib/console-roles.ts derives from USER_ROLES in @sustentus/services/shared minus sales, which is a Mongo-only system role and is never offered here. Roles are written in Clerk’s org:<role> form. Granting or revoking tenant administration is assigning or unassigning admin; there is no separate flag or claim.

Pending invitations are listed separately — email address, role and invited date, each revocable — because a Clerk organisation invite is pending until accepted and would otherwise be fire-and-forget.

The last-admin invariant. No action of this app leaves a tenant with no admin-role member: the last admin can be neither removed nor moved off admin. The count comes from a role-filtered Clerk membership query rather than the 100-row roster page, and the guard runs before any write, so a refused action leaves Clerk and the Mongo mirror both intact. It bounds this app’s actions only — the platform’s own in-tenant user management is the second door and carries its own guard. The check is not transactional: two operators removing two different admins from a two-admin tenant at the same moment can both pass it, and Clerk offers no lock to close that window.

Scoping and authorisation are unchanged from the directory. Every action calls requireViewer() and then re-resolves the tenant through tenantService.findForConsole(scopeOf(viewer), tenantId), so a partner acting on a tenant that is not theirs gets the same not-found as for one that does not exist. Outcomes travel back as fixed codes in the query string, never as echoed input. Removal removes the organisation membership only — the person keeps their Clerk account and every other tenant they belong to.

Reverting the code does not revert the writes. There is no migration, so the change is clean to roll back, but invitations already sent stay live, removed memberships stay removed and changed roles stay changed. Undoing those means the Clerk dashboard.

Plan and subscription

/tenants/:id carries a subscription card for both audiences: the tenant’s plan name, the seats it is using against the seats it is allowed, and its subscription status. It is read-only — there is no control to move a tenant between plans (see the limitation at the end of this section).

Clerk Billing is the system of record, read live on every request through clerkClient().billing.getOrganizationBillingSubscription() in apps/console/lib/console-billing.ts. Nothing about a subscription is mirrored into Mongo, and nothing is cached: a commercial state is exactly the sort of fact that is wrong the moment it is stored.

Seats come from the organisation, not the subscription. Clerk Billing carries no seat quantity, so “used” is the totalCount of a one-row membership read and “allowed” is the organisation’s maxAllowedMemberships. Counting the member page the view renders would cap every large tenant at that page size and show it comfortably inside an allowance it had blown through. Clerk writes “no cap” as 0, which the card states in words rather than printing a zero that means the opposite.

No money reaches the card, for either audience. TenantSubscription carries a plan name and a status and nothing else; fee, annualFee, amount, nextPayment and lifetimePaid are in scope only inside console-billing.ts and are never mapped out of it. The partner money boundary therefore holds by construction rather than by a per-audience branch in the view — there is no amount in the shape to leak.

Three answers, not two. A read returns subscribed, none (Clerk’s 404 — this tenant has no subscription) or unavailable (Clerk’s 403 — this instance may not read Billing). Collapsing the last two would report a paying tenant as unsubscribed the day a billing permission is revoked, so isClerkForbidden in apps/console/lib/clerk-errors.ts is deliberately kept apart from isClerkNotFound. Any other Clerk failure is a genuine fault and throws.

Billing is not switched on for this Clerk instance. It is a public beta, and until it is enabled every tenant reads Unavailable for plan and status. Seats are live and correct regardless, because they come from the organisation rather than from Billing.

Moving a tenant between plans is not possible here, and not merely unbuilt. Clerk’s backend Billing API exposes no plan-change write — its only writes are cancelSubscriptionItem and extendSubscriptionItemFreeTrial. A plan switch happens in the payer’s own checkout, which console users cannot reach: they are org-less by design and never enter a tenant organisation. Offering the move needs a writable subscription store, which is its own piece of work.

Platform markup

commercial.platformMarkupPercent is Sustentus’s platform markup on a tenant’s services revenue — a percentage from 0 to 100, decimals allowed. It is one of the registered settings on the tenant detail page’s Settings card, and it is set only from here, by staff: the registry marks it consoleOnly, so the tenant’s own generic setting writes in apps/web (create, update, remove on tenantSettingService) refuse it — creating it, renaming a row into or out of it, and deleting its row — and the onboarding blueprint cannot propose it. The one write path is setRegisteredSettings, which also enforces the registry’s bounds: a value below 0 or above 100 is refused as invalid and nothing is written. Each change lands on the activity trail as tenant.setting_changed with the key, the new value, and who made it.

Unset means no fee. The key has no registry default and is never seeded; clearing the field removes the row. There is no hidden default percentage anywhere — a reader that finds no row shows no fee rather than a £0 one. Today the markup shows on the SDM’s service-revenue figures, as a platform fee on paid revenue. It is not a go-live readiness fact: the commercial check reads the tenant’s pricing-rules flag alone, because an unset markup is a valid state the tenant’s admin cannot change.

Invoice tax rate

commercial.taxRatePercent is the tax rate on every invoice the tenant raises — a percentage from 0 to 100, decimals allowed. It sits beside the platform markup on the same Settings card and follows the same rules: the registry marks it consoleOnly, so only staff set it here and the tenant’s own setting writes refuse it; setRegisteredSettings enforces the bounds and records each change on the activity trail. Unset means no tax — the key has no default and is never seeded. The rate in force when an invoice is raised is stamped on that invoice, so a change here never restates one already raised. What the rate does to an invoice is described in invoicing.

Prompt workspace

Staff-only, at /prompts. The console is the edit surface for the runtime prompts the agentic assistant answers on — the store, the versioning and the assembly are @sustentus/services; this app is where a person touches them.

The index lists every catalogued file grouped by its ICM layer — L0 identity, L2 register contracts, L2 rail copy, L2 surface copy, L3 reference material — with whether each is running on a published version or on the body the repository ships, and whether an unpublished draft is waiting. Two of those groups are composed into no prompt at all, so editing them changes a surface the user reads rather than the instructions the model answers on: the rail files (L2/rails/<register>.md) are what the agentic app’s register rail says — its verb, its one-line stance and the prose of any standing note — and the surface files (L2/surfaces/<register>.md) are what its composer and answer headers say: the name answers are signed with, the mode note beside it, the chips above the input, the placeholder and the footer line. Their file screen says so rather than reporting them as a file no register composes yet. An opening instruction (L2/openings/<register>.md) sits between the two kinds: it reaches the model, but on one turn only — the briefing a register gives unprompted when a conversation starts (The register speaks first) — and its file screen says that too. Every figure the rail shows comes from a typed server-side read instead, which is what stops a publish relabelling a number. Beside them, and marked read-only, sit the two code-owned halves of the same picture: the rails preamble composed into every assembled prompt, and the L1 routing table saying which files each register reads, in order. Showing only the editable half would misrepresent what the assistant actually runs on.

A file’s own page carries the loop: edit the markdown or upload a .md file, save it as a draft, see a line diff against the body running right now and the fully assembled prompt the draft would produce for each register that reads it, then publish it with a note. A draft is one per file, overwritten by the next save and cleared by a publish, and nothing on the request path reads it — so a draft can never reach a live agent. Publishing saves the draft first and then attempts the publish, which is what leaves a refused body where its author can keep working on it; the validator’s reasons render from that stored draft rather than being carried back in the URL.

Publishing and rolling back both carry the published version the screen was showing, and are refused if that pointer has moved since — someone else having published or rolled back the same file while the screen was open. The refused editor keeps their draft and is shown the version that landed against the one they tried to publish. It is a distinct refusal from the validator’s: nothing is wrong with the body, only with the world it was measured against. And it is detection, not a lock — the file is never held while someone edits it, so the window is narrowed to the moment of the write rather than closed.

When the store cannot say what is live, neither screen fills itself in from the shipped seeds: seed text presented as the published body is how someone edits and republishes over a live version they never saw. A store that cannot be reached at all replaces the file list and the file screen with a notice saying so, that nothing has changed, and to try again in a moment. A file recorded as published at a version no longer in the store is the opposite case — the store is reachable and can name the problem — so that file is shown and flagged, with its history and its rollback control intact. Publishing it or rolling it back both repair it: either writes a pointer that resolves, and a file is no longer refused a publish for its own dangling pointer. What stays refused is a publish of a different file whose assembly still composes the dangling one, because that assembly cannot be built until the pointer is repaired — so one dangling pointer in a shared file like L0/identity.md, composed into every register, no longer makes every file unpublishable. Publishing is refused while either condition holds and says which it was: retrying is the next step for an unreachable store, and repairing the named file for a dangling pointer.

History shows published versions newest first with their author, time and note, one page of 50 at a time — promptversions is append-only, so the read is bounded in the query (listVersions runs through findPaginated, default page size 50, served by the { filePath: 1, createdAt: -1 } index) rather than by the data. The card’s caption says which versions are on screen (“Showing versions 1–50 of 123”) and a pager reaches the rest; a history that fits on one page is captioned as showing every version. The live version is always named, even when a rollback has put it on another page. Any version can be compared with the live one or restored — rollback resolves its target by id (readVersion), so a version on any page rolls back the same way. Restoring writes no new version — it moves the file’s published pointer — so the history is never rewritten; what records who restored what is the prompt.rolled_back entry on the activity trail below.

Staff-only three times over, and independently: the nav entry is staffOnly, proxy.ts matches /prompts(.*) and /api/prompts(.*) in isStaffRoute, and each of the six server actions opens with requireStaff() — the test conversation’s route handler makes the same live read through readStaffViewer() and answers 403, because its caller is a fetch and cannot follow a redirect. The nav is presentation — the other two are the gate. A path that is not in the code-owned catalogue is a 404, checked before any store read.

Every interaction on the file and register screens is a plain form against a server action. The one client component on this surface is the test conversation below, which streams.

One register over time

/prompts/registers/:key is the same workspace read the other way round: not one file, but one register’s whole prompt as it has changed. The key is the agent registry’s (assistant, csm, vendor, expert, sdm) — the value a run record stores in agent — and its prompt-workspace routing is looked up from it, so a register with no runs to show is a 404 rather than an empty page.

Three things, because they are one question:

  • The publish timeline — every published version of every file the register composes, newest first, with who published it, when, and their note, and a diff of any publish against the version published before it.
  • Run quality either side of each publish, and it is matched by version stamp, not by clock: every run record carries the versions that composed the prompt it answered on, so “before” is the runs stamped with the previous version and “after” is the runs stamped with the one this publish introduced. Counted for this register only — L0/identity.md is composed by every register, so a publish to it would otherwise pool all of their turns into one register’s page. Where a side has no runs the page says so; a publish nobody has answered on yet and a publish nothing has failed on are opposite facts, and 0% for both is what would let somebody sign off a bad edit.
  • Error and refusal rates by month, with the models that served those runs — so a rate that moved can be read against the publishes above it, against a change of model, or attributed to neither.

The pairing is “the version published before”, not “the version this replaced”. Restoring moves the file’s pointer and writes no version, so after a restore the two differ. The screen says which it means rather than claiming more than the data supports.

Scored evals

The same page runs a code-owned eval suite (packages/services/src/ai/evals/) against either the published version set or the set the currently saved drafts would produce — which is how a draft is scored before publishing. Four checks per case: claims present and well-formed (through isRenderableAnswerPart, the guard the chat surface itself applies), tool scope correct (the tool is one the register mounts, the prompt still names it, and the scope is one the question permits), register verb honoured, and caveats present.

Every register has a case file, and each carries a <register>-house-style case: it fails when a publish of the shared answering style (L3/how-to-answer.md) drops the rules on British English, exclamation marks or closing offers, so a style regression shows on every register at once.

Every grader is a string or shape comparison — no model call, no network, no clock — so the same version set always scores the same and a failure explains itself. Each run is recorded in agentevalruns against the version set it scored, with who ran it and whether it scored the published or the draft-applied set.

Informative, never blocking. promptWorkspaceService.publish reads no eval result; scoring publishes nothing and refuses nothing. Gating is a policy change in publish once the checks have earned trust, deliberately left as a line to add rather than pre-wired as one to remove.

Talk to it

The checks say what a prompt says; /prompts/registers/:key/test shows how the assistant behaves on it, before anything is published. It is a real, streamed, multi-turn conversation on one register’s prompt from one of the two sources the evals score — draft, the prompt the register would answer on if every saved draft were published, or published, exactly what it answers on now — assembled by the same promptWorkspaceService.assembleForSource the eval action uses, so what is scored and what is talked to never differ. The page defaults to the drafts when any file in the register has one, and says which; it is reached from the register page beside the scores and from a file screen’s draft card.

The page states what the conversation runs on and the route guarantees it. It is bound to the demo tenant: POST /api/prompts/test resolves the tenant flagged isDemo on the server, refuses unless exactly one is flagged, asserts it again with requireDemoTenant before any tool is built, and closes every tool over it with createTenantContext — the request carries no tenant field, and a tenantId planted in the body changes nothing. It honours the assistant switch like every other assistant surface, refusing with the same 503 and the same sentence. The tools are the register’s on a console caller’s terms — no persona actor and no permissions, so the sets that read as a CSM, vendor, expert or SDM mount nothing, and the page lists each set as mounted or not, computed by building them exactly as the route does. Nothing is persisted: no thread, message, run record, eval record or activity entry. The client keeps the transcript and posts it whole each turn; the route accepts user and assistant turns only (a system message is refused, as the live route refuses it), keeps only their text parts, caps the transcript at 40 messages and 60,000 characters, and composes the system prompt on the server every turn. A draft turn that would have applied no draft — the draft discarded while the page sat open — is refused rather than answered on the live prompt under the wrong label.

Two consequences are deliberate. Test turns are unmetered: with no run record they never reach /ai-spend or /agent-runs, and the gateway budget plus the transcript cap are the bounds. And the live chat route in the agentic app is untouched — the draft-reading assembly has exactly two callers, the eval service and this route, and a unit test pins that assembleForRegister resolves no draft even when every file has one.

Activity trail

Every administrative action this app performs is recorded, once, in an append-only trail: ConsoleActivity in packages/services/src/db/models/console-activity.ts, written and read through consoleActivityService from @sustentus/services/server.

An entry carries the actor’s Clerk user id, their name and email, the audience they acted as, the action, the subject tenant and its name, a one-line detail, and the time. Names are snapshotted at write time, not joined. Console people are org-less Clerk users with no row in the platform user collection, so there is nothing to populate from, and resolving each actor against Clerk would be one API call per rendered row. Snapshotting also keeps an entry readable after the person’s access has been revoked — which is exactly when someone comes looking at it.

What is recorded

The thirty-three recorded actions, and nothing else:

Action fileRecorded as
tenants/new/actions.tstenant.created
tenants/[tenantId]/actions.tstenant.partner_assigned, tenant.partner_cleared, tenant.user_invited, tenant.invitation_revoked, tenant.user_removed, tenant.user_role_changed, tenant.suspended, tenant.reactivated, tenant.deleted, tenant.profile_changed, tenant.setting_changed, tenant.sla_changed, tenant.sla_removed, tenant.role_template_edited, tenant.role_template_reset, tenant.user_permissions_edited, tenant.catalogue_created, tenant.catalogue_changed, tenant.catalogue_archived, tenant.catalogue_linked, tenant.catalogue_unlinked, tenant.assistant_disabled, tenant.assistant_enabled — the configuration edits once per key, stage, role template, user or catalogue entry that actually moved, so a save that changed nothing records nothing
tenants/deleted/actions.tstenant.recovered, tenant.removed
access/actions.tsconsole.access_granted, console.invitation_revoked, console.access_revoked
prompts/actions.tsprompt.published, prompt.rolled_back — its other three actions (saving, uploading and discarding a draft) change nothing the agents run on and are recorded nowhere
assistant-switch/actions.tsassistant.disabled, assistant.enabled — the platform-wide assistant kill switch, carrying the reason the flipper gave as the entry’s detail

Recording happens after the action’s own write succeeds, so a refused or invalid action leaves no trace and a completed one leaves exactly one entry. The write is awaited and its failure surfaces: the primary write has already committed by then, so swallowing the error would mean “done and not recorded”, with nobody told.

The three access/actions.ts entries carry no subject tenant — granting someone console access is an act against the console, not against a tenant. That makes them staff-only by construction: a partner’s read is bounded by their assigned tenant ids, and no tenant matches none. The two prompts/actions.ts entries are tenant-less for the same structural reason and gain the same property: the prompt workspace is platform-wide and holds no tenant data at all. The two assistant-switch/actions.ts entries are tenant-less for the third time over: the switch they record is the platform-wide one, and turning the assistant off for every tenant at once is Sustentus’s act rather than any tenant’s. The per-tenant half of that switch lives on tenants/[tenantId]/actions.ts above and does carry its tenant.

Who sees what

/activity lists entries newest first on the directory’s own URL contract (page, 1-based; junk reads as page 1). Staff see every entry, the tenant-less ones included. A partner sees only entries whose tenant is currently assigned to them, resolved per request from the same assignment the directory reads — so a re-assigned tenant takes its history out of the old partner’s view and into the new partner’s. Denormalising partnerId onto the entry would freeze it at write time and disagree with the directory the moment an assignment moved.

A partner is also bounded by action. They see an entry only when the console shows them the same fact somewhere else: the tenant’s lifecycle (tenant.created, suspended, reactivated, deleted, recovered) and its user management (tenant.user_invited, invitation_revoked, user_removed, user_role_changed) — nine actions. Every act behind a staff-only card on tenant detail is hidden from them: the subscription tier, the per-tenant assistant switch, the profile and settings, SLA timers, role templates and permission overrides, the catalogue, and who the assigned partner is. Those entries’ detail restates the fact, and for the tier and the assistant switch carries the staff member’s free-text reason. The rule is one total map, CONSOLE_PARTNER_VISIBLE_ACTIONS in the trail’s service, so a new action does not compile until somebody decides whether a partner reads it. Existing entries are filtered at read time; nothing is migrated.

The bound is applied inside consoleActivityService.listForConsole, not by the shell’s nav, so a partner requesting /activity directly gets the same scoped answer as one who followed the link. Staff reads declare crossTenant: true — the tenant plugin’s documented escape hatch is how a deliberate cross-tenant read announces itself. A partner’s { tenantId: { $in: [...] } } filter needs no hatch, because the plugin can see it. Its action: { $in: [...] } clause rides beside the tenant clause as a residual filter on the tenantId_createdAt index.

That scoping is not restated anywhere. consoleActivityScopeFilter and consoleActivityReadModel are exported from the trail’s service and reused by the Notification centre, which reads these same rows through these same bounds. The centre sets its own action clause, which would replace the partner’s, so it narrows a partner’s notifiable set to the partner-visible one itself (notifiableConsoleActionsFor).

Immutability

Nothing edits or deletes an entry. The service exports a writer and two readers and no mutator, no route or action offers one, and the model carries no softDeletePlugin — an audit is never edited or removed. A unit test asserts the exported surface holds no update or delete method.

The one thing that removes entries is permanent removal, which empties every tenant-scoped collection and the trail is one of them: afterwards nothing tenant-scoped about that tenant remains — its creation, its suspensions, its user changes, all gone with it. One entry survives. The tenant.removed entry is written tenant-less, snapshotting the removed tenant’s name, so the console still records that Sustentus removed a workspace and who did it; being tenant-less it is staff-only, which is the right audience for a tenant that no longer exists.

Indexes

autoIndex is off on this schema, so both indexes come from migrations rather than from model registration:

IndexServes
createdAtthe staff feed — every entry, newest first, across every tenant
tenantId_createdAta partner’s $in feed, and any future per-tenant read

createdAt deliberately does not lead with tenantId, unlike the rest of this database: the staff read of this collection has no tenant clause to lead with, and a single-field createdAt is not a prefix of the compound one. Both are created with the schema’s { locale: "en", strength: 2 } collation, named explicitly — an index created without a collation inherits the collection’s default, and autoCreate (a separate switch, still on) stamps the schema’s collation onto the collection, so leaving it unnamed would make the result depend on whether the migration or the app’s first model compile got there first.

Notification centre

The trail records; the centre tells. /activity and the dashboard feed are both places somebody has to go — a partner learns a tenant of theirs was suspended by opening the console and looking. The notification centre is the same information arriving instead: a bell in the shell header carrying an unread count, a popover previewing the newest unread, and /notifications as the full list.

A projection of the trail, not a second record

There is no notification collection and no fan-out write. A notification is a console-activity entry, seen by somebody who is in scope for it. The centre — consoleNotificationService in packages/services/src/db/services/console-notification/ — reads that one collection with three extra predicates over the trail’s own scoped filter, and lays per-person read state on top:

PredicateWhy
action in the notifiable setseven of the trail’s twenty-four actions
the trail’s own audience scopeidentical to what /activity shows
actorClerkUserId not the viewernobody is told of their own action

Three properties follow from that and are not maintained by anything:

  • The two views cannot disagree. Nothing writes a notification, so nothing can fail to write one, and no reconciliation job is needed to keep a second collection in step with the first.
  • Removal is inherited. Permanently removing a tenant empties its trail entries, so its notifications go with them. The one tenant-less tenant.removed entry the trail keeps back survives as a staff notification naming the removed workspace — same row, same rule.
  • Marking read touches the trail not at all. Read state is a separate per-person collection, so the audit stays append-only and the same entry can be read by one person and unread by another.

Which entries are notifiable

Seven of the twenty-nine recorded actions — the ones about a workspace existing, its lifecycle moving, or somebody’s power over it changing:

tenant.created · tenant.suspended · tenant.reactivated · tenant.deleted · tenant.recovered · tenant.removed · tenant.user_role_changed

The set is CONSOLE_NOTIFIABLE_ACTIONS, declared as a total Record over ConsoleActivityAction in the shape CONSOLE_ACTIVITY_LABELS already uses. A new trail action therefore stops the file compiling until somebody decides whether it is notifiable — the twenty-two that are not (partner assignment and clearing, the three user-management actions, the three console.* access actions, the two prompt.* ones, the seven tenant-configuration ones and the five tenant.catalogue_* ones) are false there rather than absent.

tenant.profile_changed, tenant.setting_changed, tenant.sla_changed, tenant.sla_removed, tenant.role_template_edited, tenant.role_template_reset, tenant.user_permissions_edited and the five tenant.catalogue_* actions are silent by decision, not omission: the console trail is the whole audit of a staff configuration edit — role templates, permission overrides and catalogue entries included — and the tenant is deliberately told nothing about one. The staff actor is captured on the tenant’s own audit arrays (RoleTemplate / UserPermissionOverride) as a snapshot alongside the app-user by reference, so the tenant’s in-app audit views render the staff member’s name without the console ever creating a platform user row for one of its own people.

There is no plan-change notification because there is no plan-change action to project — see Plan and subscription above. When plan movement becomes possible it is one more trail action and one more true in that map.

Who is told

Scope is the trail’s scope, resolved live per request, and it is the trail’s own code rather than a copy of it: consoleActivityScopeFilter and consoleActivityReadModel are exported from db/services/console-activity/ and used by the centre. A partner’s { tenantId: { $in: [...] } } bound is the whole of partner authorization on this collection, so a second copy would be a second place for it to widen.

Both of those functions match scope.audience === "sustentus" positively rather than testing for !== "partner". The union has two members, so the forms are equivalent today — but only the staff audience can then reach the unbounded filter and the crossTenant escape hatch, and a third audience added later is bounded by default instead of inheriting staff’s view.

actorClerkUserId: { $ne: <viewer> } is applied in the query, not as a filter over the results, so the list, the unread count and the page count all agree about which rows exist.

Read state

Two collections in packages/services/src/db/models/console-notification.ts, neither carrying a tenantId — so neither takes tenantPlugin, neither needs a demo-reset policy classification (policy.test.ts classifies schemas that carry tenantId), and neither is reached by tenant-purge:

CollectionShapeWritten by
console-notification-cursorone per person: clerkUserId, allReadBeforeMark all read
console-notification-readone per person per entry: clerkUserId, activityIdMark read

A notification is read when its entry predates that person’s cursor or its id has a read row. Mark all read is O(1) — stamp the cursor, then delete that person’s read rows, all of which name entries the new watermark now subsumes. The read rows are bounded because markRead performs the same compaction the moment nothing is left unread: without that, somebody who only ever marked rows individually would never reach the operation that prunes them.

Both writes are upserts against a unique index, which is not atomic — two submissions of the same mark race, and the loser’s insert raises a duplicate key. Both catch it through the house isDuplicateKeyError, because the operation had in fact succeeded.

markRead re-resolves the entry through the same scoped, notifiable, not-your-own read before writing anything, so a read row can only ever exist for a notification that was genuinely the caller’s. Server actions are dispatched by action id and can be posted from any path; that re-resolution is the boundary, not the page that renders the button.

Indexes come from a migration with autoIndex off, following the console-activity precedent, and name the schema’s { locale: "en", strength: 2 } collation explicitly for the same reason the trail’s do. No index was added to console-activity: staff sort on createdAt and a partner matches tenantId then sorts, which the trail’s two existing indexes already serve; the notifiable action, the actor exclusion and the read-row exclusion are residual filters over a set those have already bounded, at a volume of a handful of administrative actions a day.

Where a notification goes

Derived from the action alone, with no extra lookup per row (notificationHref in apps/console/lib/console-notification-links.ts):

ActionDestination
tenant.removednone — the entry carries no tenant and the tenant is gone
tenant.deleted/tenants/deleted for staff; unlinked for a partner, who has no deleted-tenant surface
everything else/tenants/:id, itself scoped, so a link a reader should not be able to follow behaves exactly as a typed URL does

That file imports only a type, so it holds no server dependency and is unit-tested directly.

What the centre does not do

  • It publishes nothing realtime. No Ably message — packages/services/src/notifications/ is the tenant platform’s per-resource notify path and is not involved here. The centre itself is an in-app projection for people signed into the console. The same events are now delivered by email, by a separate path that reads this same set — see Email notification below.
  • Marking read never removes a notification. The row stays, shown as read, because the trail entry behind it is the audit.
  • Opening one does not mark it. Reading is an explicit act.
  • Nothing is realtime. The count and the preview are read in the shell’s server component when the page renders. Both mark actions revalidatePath("/", "layout"), so the count is right after either — but App Router does not re-render a shared layout on client-side navigation between the routes under it, so a count is otherwise as of the last full load. Moving the header into a (console)/template.tsx is what would change that.
  • The mark-all-read watermark is global, while scope is resolved live. A tenant assigned to a partner after they last marked all read arrives with its history already behind their cursor, so those entries are in their centre — the list is correct — but never raise the badge. Fixing that means a per-scope watermark.

Email notification

The centre tells whoever signs in. Email tells the people who will not — a tenant’s own administrators, who have no console account at all, and an operator who is not looking today.

The event set is the centre’s, not a second list. CONSOLE_EMAIL_MATRIX (packages/services/src/notifications/console/audience.ts) is a total Record over ConsoleActivityAction saying which of three audiences each action reaches, and a unit test asserts both directions against CONSOLE_NOTIFIABLE_ACTIONS: every notifiable action emails somebody, and nothing the centre stays silent about sends mail. The two channels cannot drift.

ActionTenant adminsPartnerStaff
tenant.created tenant.suspended tenant.reactivated tenant.deleted tenant.recoveredyesyesyes
tenant.removedyes—yes
tenant.user_role_changed—yesyes
the twenty-two non-notifiable actions———

tenant.removed reaches no partner because the entry it projects carries no tenant — the same reason it is absent from a partner’s centre. Its tenant-admin recipients are read before the Clerk organisation is deleted and handed to the send, because afterwards there is no membership left to read.

Two vocabularies

A tenant administrator is not a console user, and their mail says so: no console link, no console words, no actor name — they are told what happened to their workspace, with support@sustentus.com as the contact and the Reply-To. Partner and staff mail is operator-voice and links back to the tenant at the same destination notificationHref gives the centre’s row, so an email and a notification about one event land on the same page. With CONSOLE_APP_URL unset the operator mail sends linkless rather than carrying a link to nowhere.

Nobody is told about their own action — matched on the actor’s Clerk id or their address, because a recipient can reach the fan-out without an id (SYSTEM_ADMIN_EMAIL is a bare address, and so is a pending invitation).

Recipients

Resolved in apps/console/lib/console-event-email.ts, because every one of them is a Clerk read — which is this app’s dependency, not the services package’s:

AudienceResolved from
Tenant adminsorg:admin memberships and pending org:admin invitations on the organisation
Partnerthe tenant’s assigned partnerId, or nobody when it has none
Staffthe console invitation roster, plus SYSTEM_ADMIN_EMAIL when it is set

The invitations half is not a nicety: a new tenant’s organisation is created empty — console people are org-less, so there is no createdBy to make a first member — and its first administrator exists only as an invitation. Reading memberships alone would find nobody at precisely the moment the “your workspace is ready” mail is meant to send.

The roster half is likewise: it is derived from invitations this console created, so it cannot see somebody granted access directly on an existing account — the bootstrapped admin especially. The env address closes that without a second mechanism, and the fan-out deduplicates by address, so naming somebody who is also on the roster costs nothing.

Failure is recorded, never raised

The send runs in after(), after both the administrative write and the trail entry have committed, and nothing in it can affect the action that caused it: every path resolves, the after() call is itself wrapped, and a total failure is a set of rows rather than an error an operator sees. A suspension that succeeded stays succeeded, and the fact that nobody was told is written down instead of lost.

console-email-delivery-log is that record — append-only, failures only, bounded by a 30-day TTL index created by migration with autoIndex off. It is its own collection rather than a row in the platform’s notification-delivery-log, whose tenantId is required by tenantPlugin and which the tenant-less tenant.removed cannot supply, and whose recipients are platform user rows console people do not have. That separation is about what is being recorded, not about reach: the console also reads the platform’s log, cross-tenant, through consoleNotificationDeliveryLogService — see Fleet delivery below.

ReasonMeans
not-configuredRESEND_API_KEY is unset in this environment
transport-errorthe provider rejected or threw
recipient-missinga resolved recipient carried no address
recipients-unresolvablethe Clerk read for that audience failed

Deliberate non-delivery is never a failure. A tenant with no partner, or an audience the actor was filtered out of, produces no row — an empty audience is a fact, an unreadable one is a fault, and only the second is logged.

Staff read the failures at /notifications/delivery — gated twice, by the proxy’s staff matcher and by requireStaff() on the page — reachable from a staff-only nav entry so it can be opened to confirm health, not only to diagnose breakage. The dashboard reports the trailing 24 hours in both states — an alert when sends have failed, distinguishing a missing provider key from per-send faults, and an all-clear when none have. Both reads declare crossTenant: true: the question is whether console mail is getting out at all, which is about the estate, and the tenant-less removal rows no tenant filter could reach.

Fleet delivery

Staff-only, at /notifications/fleet. Where the section above is the console’s own mail, this is the platform’s notifications — the in-app rows, the Ably pushes and the Resend emails every tenant’s people are meant to receive.

Those are fire-and-forget by design: createNotification never throws to its caller, and a dead Ably key, an unregistered template or a missing address degrades to a warn-and-continue no-op. notification-delivery-log already recorded each failure, but the only surface onto it was each tenant’s own /admin/notifications page — a page tenant admins rarely open and staff cannot open at all. A platform-wide breakage therefore surfaced as N separate tenant logs nobody was reading. This page adds them up.

Read-only, and cross-tenant by construction. The reads live in consoleNotificationDeliveryLogService, a console-scoped entry point beside the tenant-scoped notificationDeliveryLogService rather than a flag on it, so the exemption never reaches the surface tenant admins use. Nothing here writes: the only writer of that collection stays notifications/core/record-failure.ts, and the fix for any row happens where the failure points — a template, a key, a recipient.

The three reads declare their exemption in two ways, and the difference is the point:

ReadShapeHow the exemption is declared
recentFailuresthe capped row list.setOptions({ crossTenant: true }) — tenantPlugin hooks find and honours it
failureBreakdownone $group behind the fleet page’s summary line and filter options, tenant names resolvednamed in a comment at the pipeline — tenantPlugin does not hook aggregate at all, so there is no flag to declare
failureSummaryone $group behind the dashboard alert — total, tenants affected, misconfigured; no tenant names, no option liststhe same comment at its own pipeline; it shares the window $match and the misconfigured rule with failureBreakdown, so the two counts agree

No new index and no migration: the unscoped window read is served by the collection’s existing createdAt_ttl index traversed in reverse, a tenant-filtered one by tenantId_createdAt, and the same TTL is what keeps the grouped read cheap.

What the page says out loud. It covers seven days, matching the tenant-side page so the two never disagree, and states the log’s 30-day retention beside it — an empty page is not a clean month. It is capped at the 50 most recent, like /agent-runs and for the same reason: a second page of fleet failures would mean something is wrong at a scale a pager does not help with. The summary counts and the filter options are the whole estate’s even when the list is narrowed, because the options are built from that same read, and the page says so rather than leaving a reader to trip over it.

Filters — tenant, notification type and channel — are applied in the read, not the render, and each is validated against the thing that makes it meaningful: a tenant by shape, so a tenant that failed nothing keeps its filter and gets the empty list that is the true answer; a channel against the model’s closed union, for the same reason; a type against the window’s own types, because the stored type is an open string where an unmatched value is far likelier a typo than a quiet notification.

The recipient is deliberately not resolved here. The tenant-side log names them because it is already inside that tenant; doing it on a fleet view would widen a triage read into every tenant’s user records for no diagnostic gain. Staff follow the tenant link — present on every row, and rendering a deleted tenant unlinked under its id rather than dropping it, because those failures were real and hiding them would stop the totals adding up. A tenant’s own slice is reachable the other way too, from a staff-only card on /tenants/:id that reports its seven-day count and links here pre-narrowed — that card reads the tenant-scoped service, since there a tenant leads the query and a cross-tenant flag would misdescribe it.

Assistant health

Staff-only, at /assistant-health. The agentic chat route (apps/agent/app/api/chat/route.ts) degrades rather than fails: when the conversation store is unreachable the turn is still answered and no conversation is stored for it, because a database blip should cost a user their history rather than their assistant. The run record is still written, marked persistence: "unsaved", so the turn’s spend is metered even though its conversation was lost. That choice left no trace beyond a console.error, so an outage looked from the outside like a product that works. This page is the trace.

agent-degradation-log is the record — append-only, calls that did not happen only, bounded by a 30-day TTL index created by migration with autoIndex off, exactly as console-email-delivery-log above. One row per store call the route swallowed:

FieldMeans
operationwhich call was lost — thread read/create, message writes, history read, run record, or the register rail’s read
outcomewhat it cost: served-unsaved, history-lost, write-lost, or rail-lost
causefailed (the call threw) or refused (the store answered and declined)
errorCodethe error’s name only — never the driver’s message text. Present exactly when cause is failed
firstOfTurntrue on the one row per turn that tipped it into degradation

One operation here is not a conversation-store call. The agentic app’s register rail is read on every page render, and a read it loses is written as rail read / rail-lost: the page still serves the register’s verb card and its published copy, and only the figures are missing. It shares this vocabulary rather than getting a collection of its own, so “is the assistant losing reads” is answered from one place. It is also the one operation that never carries firstOfTurn — that field counts degraded turns, and letting a per-render read claim it would make the headline figure count navigations instead.

cause is a third axis, not a fourth outcome. A refused write costs the turn exactly what a failed one costs (write-lost), so the two are orthogonal; folding them together would need a *-refused twin of every outcome. The distinction earns its place on the deploy-to-migration window: appendMessage matches on nextSequence: { $exists: true }, so a thread the agent-thread-next-sequence migration has not reached is refused rather than written at a position that was never reserved. Production migrations run behind a required reviewer, so between a deploy and its approved migration the assistant answers normally while conversations stop recording — a run of refusals on this page is what that looks like, and the page says so. Rows written before the field existed carry no cause and are read as the failures they were.

firstOfTurn is the counter. A turn that loses its thread read, its history read and its run record is one degraded turn, not three, so “how often is the assistant serving turns it cannot save” is countDocuments({ firstOfTurn: true }) over a window rather than a row count.

Recording can never cost a turn, and costs one write. The work is handed to after() and swallows everything, including its own failure — which matters more here than for the email log, because it goes to the very database that is already failing. A turn’s rows are collected in memory and written as a single insertMany after the response, so the model — and therefore connectDB(), and therefore a server-selection timeout — is resolved once per degraded turn rather than once per lost call. firstOfTurn is the batch’s first row, decided before anything is written rather than raced for between writes. Do not reintroduce a per-row reporter: it turns the cheapest path in the route into a multiplier on connection churn exactly when the pool can least absorb it. tenantPlugin hooks queries and not insertMany, so every document sets tenantId explicitly.

The collection is the readable record; a drained log line is the durable one. The collection lives in the database the degradation is usually about, so a total Mongo outage takes it with the store calls it reports. recordAgentDegradations therefore mirrors every batch to the agent app’s standard error before it resolves the model — before connectDB(), before server selection, before any timeout — as one line of JSON whose first key is the marker sustentus.agent-degradation. The line carries exactly what a row carries (tenant, operation, outcome, cause, the error’s name, firstOfTurn) and is built field by field so it cannot carry anything else. A window /assistant-health shows nothing for is reconstructed from that mirror; the page says so, always, because a gap and an all-clear look identical.

The sink is a Vercel log drain on the agentic-interface project, and that was chosen over PostHog and over Sentry: no vendor SDK, no API key, no dependency and no turbo.json → globalEnv entry. What it buys is durability and only durability — no alerting, no grouping, no dedupe, no retention policy. Nobody is paged by a mirrored line.

Filter to the marker — at the destination, not at the drain. Draining the function log means the log leaves the server, and the console.error lines the chat route writes beside each degradation carry the raw driver error, hostnames and credentials included. The mirror line is redacted by construction; those lines are not, and are deliberately left that way — the raw error next to the stack is what a developer reads in Vercel’s own log view.

A Vercel log drain has no per-line content filter: it selects sources (lambda, edge, build, static), environments, projects and a sampling rate, and forwards whole lines. So the filter is the destination’s job — an ingest rule or pipeline processor that keeps only lines whose first key is sustentus.agent-degradation and drops the rest. Point the drain at a store that can do that, and configure the rule when you configure the drain; a drain into a bucket that keeps everything is the thing this warns against.

The reporting path’s own two failure lines were reduced to error names for this reason and need no filtering: recordAgentDegradations’ warning (which fires one line after the mirror, on the very outage the mirror exists for) and the agent reporter’s guard in apps/agent/lib/degradation-reporter.ts. Only the chat route’s per-call lines are still raw.

Staff read it at /assistant-health, gated twice like the email log — the proxy’s staff matcher and requireStaff() on the page — reachable from a staff-only nav entry so it can be opened to confirm health, not only to diagnose breakage. Both reads declare crossTenant: true: the question is whether the assistant is persisting at all, which is about the estate. The tenant name on each row is resolved at read time rather than denormalised at write time, because a name lookup on a failing store would fail with the rest of it.

Agent runs

Staff-only, at /agent-runs. /assistant-health above answers whether a turn was saved; this page answers whether it worked. Before it, a failed or refused turn produced a console.error on a serverless log nobody reads and a run record no screen rendered — so the first signal was a user complaint.

Nothing new is written for it. The run record (agentruns) already carries the register, the model, the outcome and its error code, the tool calls with their bound tenant and effective scope, and the prompt version set behind the answer; this page reads it. Failures first and by default, with every turn one click away for the times the question is “what does normal look like”, filterable by register and by month.

Each row expands the same TraceRail the conversation surface uses, and lists the turn’s prompt version set with a link per file into the prompt editor’s compare view — so the text behind a failed answer is one click from the failure rather than a search. A file served from its repo-shipped seed is labelled as the seed rather than linked to a version page that would 404.

The tenant shown against a tool call is the call’s own binding, never the run’s. A tool is closed over one tenant at build time, and this is the screen built to audit that — so a call bound somewhere other than the run’s tenant renders as its own id rather than under the run’s name.

Capped rather than paginated, like the assistant-health log and for the same reason: it is a diagnosis aid, and a second page of failures would mean something is wrong at a scale a pager does not help with. The population is stated on the page — including how many of the month’s runs carry no version stamp, because those predate it and belong to no version.

The reads are unscoped by design and staff-only for it: a prompt version has no tenant, so “how did runs on this version go” is cross-tenant by construction. The one find declares crossTenant: true; tenantPlugin does not hook aggregate at all, so the pipelines carry the staff gate at their callers instead — proxy.ts’s staff matcher and requireStaff() on the page.

Home dashboard

/ is the console’s landing surface, scoped to the viewer’s audience:

  • Total organisations — every live tenant in scope: active + suspended, exactly what the directory lists.
  • Active organisations — those in the active lifecycle state, so the gap between the two numbers is the suspended ones. Deleted tenants are in neither count, and recovering one returns it to both.
  • Recent activity — the newest entries from the same scoped read /activity uses, rendered by the same component, so the two surfaces cannot describe an entry differently.
  • Email delivery — staff only, in both states: the alert when the trailing 24 hours contain a failed send, and an all-clear when they do not. The all-clear reports the absence of failures and deliberately does not claim mail is getting through: successful sends are never logged, and both counts in failureSummary() come from failure rows, so an environment with no provider key and no administrative actions in the window produces the same zero as a healthy one. It names its own 24-hour window because /notifications/delivery lists 30 days. The summary read is skipped entirely for a partner rather than merely hidden, so a partner never causes an estate-wide count.
  • Fleet delivery — staff only, built to the same idiom and for the platform’s notifications rather than the console’s own mail: both states, the same any-failure-in-24-hours threshold, the same separation of a missing provider key from per-message faults, and the same refusal to read an all-clear as proof of delivery. What makes it the fleet signal is the tenant count beside the total — one tenant failing is that tenant’s problem, the same count across a dozen is a platform breakage. Its read is skipped for a partner too.

The counts are countDocuments over the console-scope filter builders, extended with a lifecycle selector. They leave the soft-delete clause to softDeletePlugin, which hooks countDocuments as well as every find* and adds its live clause as an $and entry — beside the lifecycle clause, not merged into it. The one console filter that still states the clause itself is the single-tenant id filter, because it also feeds Tenant.updateOne and the raw collection, which the plugin never hooks.

There is no platform-health or uptime figure — that needs a real definition before it is worth printing one.

Data

The console reads MongoDB through @sustentus/services/server, so it needs MONGODB_URI and MONGODB_DATABASE_NAME in every Vercel scope it is deployed to. Outbound email needs RESEND_API_KEY and EMAIL_FROM (the bare verified sender address; the display name is applied in code), CONSOLE_APP_URL for the operator mail’s link, and optionally SYSTEM_ADMIN_EMAIL. These belong to the tenant-management Vercel project specifically — being set on web does nothing for the console. With the key unset the console still works: every intended recipient is recorded not-configured and the dashboard says so. MONGODB_URI is required and throws when absent; MONGODB_DATABASE_NAME is optional to the driver and silently falls back to the database named in the URI path, so it should be set explicitly rather than inferred.

Running it

pnpm --filter @sustentus/console dev # port 3006
Last updated on