Tenant management console
The console is the Sustentus-side operational surface: the app the Sustentus team and its partners
use to administer tenants, separate from the tenant-facing platform in apps/web.
Overview
Package name: @sustentus/console
Framework: Next.js 16 (App Router, Turbopack)
Port: 3006
Path: apps/console/
Vercel project: tenant-management
Who it is for
The console has two audiences, and they are not the platform’s six roles. A platform role (vendor, customer, expert, CSM, SDM, admin) is scoped to a tenant and governed by the permissions registry. A console audience is scoped to Sustentus:
| Audience | Who | Sees |
|---|---|---|
sustentus | The Sustentus team | Every live tenant; can invite and revoke access |
partner | A vendor partner | Only the tenants assigned to them |
Access model
Console users are org-less Clerk users carrying a publicMetadata marker:
{ "consoleAccess": { "audience": "sustentus" } }parseConsoleAccess (in @sustentus/services/shared) is the one reader of that shape. Three
properties follow from it:
-
Deny by default.
proxy.tsgates every route the app will ever have, not a chosen segment. A signed-in user with no marker reaches/no-accessand nothing else. -
The marker is read live from Clerk on every request, not from a session claim — so clearing it denies the very next request rather than waiting for a session to expire. The Clerk read behind
readVieweris wrapped in Reactcache, so it is one call per request however many components ask. -
The proxy gates console access, not every staff-only capability. Its staff-matched routes are
/access(.*),/tenants/deleted(.*),/notifications/delivery(.*),/notifications/fleet(.*),/prompts(.*),/assistant-health(.*),/agent-runs(.*),/assistant-switch(.*)and/api/prompts(.*); the rest are open to both audiences, because partners legitimately read the tenants assigned to them. A staff-only action on such a route carries its ownrequireStaff(), and that call is the whole boundary — server actions are dispatched by action id and can be posted from any path, so route matching never constrains them. -
Staff answer the email-code gate first. After the marker,
proxy.tsasksdecideConsoleGate(lib/console-gate.ts) whether a Sustentus staff member holds a gate pass issued to their own primary Clerk email; without one, every page is rewritten to the code screen at/gate(the address bar keeps the page asked for, which loads once the code is answered) and every/api/**path answers a bare 404. Partners never meet it. It is the demo hosts’ gate — the same roster, secret and screen (@sustentus/services/shared→access-gate), so one answered code opens the console, demo and explore (Demo environment → One code opens console, demo and explore). The console mails a code only to the signed-in staff member’s own address, and logs a warning naming their Clerk user id when that address is not on the roster. WithDEMO_ACCESS_ROSTERorDEMO_ACCESS_SECRETunset or malformed — a preview, say — staff keep the Clerk login alone and the proxy logs a warning once per instance; production needs both, with the platform’s secret value, plusDEMO_HOSTS. The proxy is not the only line: the viewer checks every page, route handler and server action calls (requireViewer,requireStaff,readStaffViewerinlib/console-access.ts) take the samedecideConsoleGatedecision on the same inputs, so a staff member without their own pass is sent to/gateby a page or action and refused by a route handler even on a path the proxy did not run on. Only the code screen and its two routes (/gate,/api/gate,/api/gate/request) read the viewer before the gate, throughrequireViewerBeforeGateandreadStaffViewerBeforeGate.
The only routes reachable without a marker are /sign-in, /accept, /api/health and
/api/access-gate/hop — the last only sets a pass carried in from a demo host, and whether that
pass opens the console is decided by the gate above. /accept
redeems a Clerk invitation ticket and is the app’s sole sign-up surface; it refuses anyone without a
ticket. /api/health is the probe health-check.sh reads after every merge — matched exactly, it
answers only a status word from a MongoDB ping (200 or 503).
Routes
| Route | Audience | What it does |
|---|---|---|
/ | Both | Dashboard: tenant counts and recent activity |
/activity | Both | The activity trail, newest first, scoped to the viewer |
/notifications | Both | The notification centre — the notifiable trail entries in scope, with read state |
/notifications/delivery | sustentus | Recent email delivery failures, newest first |
/notifications/fleet | sustentus | Platform notification delivery failures across every tenant, newest first, filterable by tenant, notification type and channel |
/assistant-health | sustentus | Store calls the agentic assistant could not make, newest first |
/tenants | Both | Searchable, server-paginated tenant directory, filterable by go-live readiness and setup state |
/tenants/new | Both | Create a tenant and invite its first administrator |
/tenants/:id | Both | Tenant detail, its users, its subscription, its go-live readiness, and — staff only — its configuration (profile, registered settings, the six per-stage SLA definitions, and the Roles and access panel for role templates and per-user permission overrides, and the Catalogue panel for its services, products, skills and industries), its partner and lifecycle controls |
/tenants/deleted | sustentus | Deleted tenants, with the recovery window remaining on each |
/tenants/deleted/:id/remove | sustentus | The typed-name confirmation for permanent removal |
/ai-spend | Both | Metered AI spend per tenant for a chosen month, with the population, period and basis stated beside it, and recent generation ids for a manual reconciliation against the gateway |
/access | sustentus | Invite and revoke console access |
/prompts | sustentus | The prompt workspace as its ICM tree, plus the code-owned rails and routing read-only |
/prompts/registers/:key | sustentus | One register over time: its publish timeline, the run quality either side of each publish, its error and refusal rates by month, and its scored evals |
/prompts/registers/:key/test | sustentus | A real, streamed conversation on one register’s prompt — every saved draft applied, or what is live — bound to the demo tenant and saved nowhere |
/api/prompts/test | sustentus | POST — the route the test conversation streams from; gated by the proxy and by its own staff read, answering 403 |
/prompts/* | sustentus | One workspace file: edit or upload it, diff it, preview what it assembles to, publish it, roll it back |
/agent-runs | sustentus | What the assistant’s turns actually did — failures first, with the register, the model, the outcome, the tool calls and the prompt version set behind each |
/assistant-switch | sustentus | The platform-wide assistant kill switch, and the tenants currently switched off on their own |
/no-access | — | Where a signed-in user with no marker lands |
Tenant directory
/tenants lists live tenants by name, 25 to a page, with a case-insensitive substring search and
two filters — go-live readiness and setup state (see Fleet readiness below).
/tenants/:id shows the tenant’s identity fields, derived status and setup state, its go-live
readiness, and its user list read from Clerk organization memberships (capped at 100 — the page
says so when the cap is hit, rather than reading as complete). That list is operable: see
Tenant user management below.
Scoping is applied inside the query, never after the read. The rules are pure functions in
packages/services/src/db/services/tenant/console-scope.ts:
buildConsoleTenantListFilter(scope, search?, onboardingStatus?) // → the list filter for this audience
buildConsoleTenantIdFilter(scope, tenantId) // → the detail filter, or null if malformedA partner’s filter carries their partnerId; a staff filter carries no partner clause; both
exclude soft-deleted tenants. Because the bound is in the filter, a tenant that is not assigned to
the requesting partner, a tenant id that does not exist, and a malformed id are indistinguishable
to the caller — all three are the same not-found. Nothing on a partner’s directory reveals that
other tenants exist: no global count, no pagination past their own set.
partnerId is declared on the Tenant schema and indexed by migration, and written from the tenant
detail view — see Partner assignment below.
Fleet readiness
The six go-live checks — minimum catalogue, SLA, service model, integrations, roles, commercial —
are scored across the fleet, not re-derived here. The computation is
computeTenantReadiness in
@sustentus/services/shared, the same function the tenant’s own admin dashboard calls, so the
console cannot come to disagree with the dashboard that tenant’s admin is reading.
The directory’s Go-live column names each tenant’s failing checks rather than counting them —
the point of a fleet view is to say what to chase. apps/console/lib/console-readiness-gaps.ts
maps each unmet check to where it is fixed: staff get an in-page anchor to the panel on tenant
detail that clears it (Catalogue, SLA, Roles and access) — the integrations and pricing gaps name
the tenant-side page instead, since no console panel connects an integration or sets a list
price — and a partner — who sees the
same readiness but holds none of those panels — gets the tenant-side page named instead. The
catalogue gap resolves from the gap’s first shortfall: staff land on that section of the Catalogue
panel (Products, Services or Skills — each section has its own anchor, from
lib/console-catalogue-anchors.ts), a partner is told the matching /admin/settings/* page. The
Skills section writes each skill with the services it is linked to (at least one) through
skillService, so a skills-only shortfall is cleared from the console.
Integrations have no console panel, so that gap names /admin/integrations for both audiences.
Only one of the two filters is a query. Setup state is a stored field
(tenant.onboarding.status), so it goes into buildConsoleTenantListFilter and the pager’s total
stays the size of the filtered list; a tenant whose onboarding subdocument is absent reads as
not_started there, exactly as it does everywhere else. Readiness is derived from four other
collections and cannot be sorted or skipped on, so the readiness filter reads the whole scoped list
(bounded by CONSOLE_TENANT_SCAN_LIMIT, 500), scores it in the same four reads, filters, and pages
in memory — the cost is the wider read, not more of them. The page says so when that bound is hit,
rather than truncating silently.
The scope rides in the service call on both paths, so a partner’s list is bounded before readiness is computed: the filtered directory can no more reach an unassigned tenant than the unfiltered one can. And readiness never blocks either page — it is derived from collections these views did not previously depend on, so a failure leaves the tenants themselves rendered and says the readiness is unavailable.
Tenant lifecycle
A tenant is created, suspended, ended and — eventually — erased from this app. The states live in
an optional lifecycle subdocument on the Tenant model:
{ status: "active" | "suspended" | "deleted", changedAt, changedBy, deletedAt? }Absence reads as active. Every tenant that predates the feature has no lifecycle field and is
active by that fact, so the migration backfills a stamp for the record rather than to make the
reads work.
Who holds what
Creating is the one lifecycle action both audiences hold. /tenants/new takes a name and a
first-administrator email, creates the Clerk organisation, then the Tenant document, then sends the
invitation — Clerk first, so a failure part-way leaves no Tenant document whose organisation does
not exist. A partner’s tenant is assigned to them in the same flow, through the same
tenantService.setPartner staff assignment uses; without that it would be a tenant they cannot see.
Suspend, delete, recover and permanent removal are requireStaff(), without exception — a
partner administers their clients, they do not decide whether a client still has a workspace. The
route gate above covers /tenants/deleted(.*), but /tenants/:id is deliberately open to both
audiences, so the suspend and delete actions carry their own check and that check is the boundary.
What each state means
- Suspended — the tenant stays in the directory for both audiences, badged. Its people cannot
reach the platform (see Enforcement in
apps/web). Reactivating is one click and restores nothing, because nothing was removed. - Deleted — soft. The tenant leaves both directories, its people are turned away, and it appears
on the staff-only
/tenants/deletedlist with the days left in a 30-day recovery window. Nothing has been destroyed: recovery returns it to active with its organisation, members and data intact. Past 30 days the page offers no Recover control, but the tenant is still there — nothing removes a tenant unattended. - Permanently removed — irreversible. The Clerk organisation is deleted, every tenant-scoped collection is emptied of that tenant’s rows — the activity trail among them — and the Tenant document is dropped. One trail entry survives, tenant-less: see Immutability below.
lifecycle.status: "deleted" and the schema’s isDeleted flag are written and cleared together,
by one function — buildLifecycleUpdate in
packages/services/src/db/services/tenant/lifecycle.ts. They cannot disagree because nothing else
writes either of them, and a unit test pins that.
Reading a deleted tenant
softDeletePlugin hooks pre(/^find/) and ANDs a live-only clause onto every model query, so a
deleted tenant is invisible to Tenant.find — which is the plugin working, not a bug to route
around. The deleted list, the recovery read and the removal therefore go through the raw driver,
under the plugin. Two consequences worth knowing:
- Those reads pass the schema’s
{ locale: "en", strength: 2 }collation explicitly. The driver handle knows nothing of the schema, and a string comparison issued under the collection’s default collation cannot use a collated index —clerkOrgId_1among them. buildConsoleDeletedTenantFiltertakes noConsoleTenantScopeargument at all. There is no partner shape to construct, so a partner-visible deleted read is not something a call site can build wrong.
Permanent removal
The confirmation is the tenant’s name, typed exactly, and it is checked server-side against the record — not against a hidden field, which would let anyone posting the action supply both halves of their own check.
The order is the Clerk organisation first, then the data and the Tenant document together. Identity
leads because it is the half that can fight back: while the organisation lives, a member signing in
still carries its orgId and would have upsertFromClerk write a fresh Tenant document over the
purge in progress. The cost is that an interruption leaves a Tenant document with no organisation —
which recovery detects and refuses, rather than handing back a workspace nobody can sign in to.
Erasure is policy-driven, not a hand-kept list. tenant-purge was extracted from the demo-reset
service so that both empty every tenant-scoped collection from the same lists the existing drift
test already guards. A collection added to the platform is covered by both, or by neither.
upsertFromClerk refuses a deleted tenant rather than resurrecting one. The plugin hides the
row from that method’s own findOneAndUpdate, so without the guard the first sign-in by one of its
users would silently insert a second Tenant document for the same clerkOrgId.
Enforcement in apps/web
The platform’s proxy redirects any authenticated request whose tenant is suspended or deleted to a
public /workspace-unavailable page. It runs before the route policy, because the question is
whether the workspace may be used at all — not which of its pages this role may see. The page names no cause: a suspension is a commercial or administrative matter
between Sustentus and the account, and a customer-facing page is not where that gets explained.
The gate fails open on a read error, logged. It is the one read here that cannot be narrowed to a role or a path, so letting it throw would turn a database blip into an error on every page of the platform at once — to defend a state that is administrative rather than a security boundary, and with the database down there is nothing behind the gate to reach anyway.
Partner assignment
/tenants/:id carries an Assigned partner panel, rendered and operable for sustentus only.
Staff set, change, or clear the tenant’s one partner from it — the write that makes a partner’s
scoped directory non-empty.
- The assignment is a field, not a membership: the optional
partnerId, holding the assigned partner’s Clerk user id. One partner per tenant is the field’s cardinality, not a rule that has to be policed. Clearing$unsets it, so an unassigned tenant reads the same whether it was never assigned or was cleared. - One shared write.
tenantService.setPartner(tenantId, partnerId | null)resolves the tenant throughbuildConsoleTenantIdFilter({ audience: "sustentus" }, tenantId), inheriting the live-only guard and the malformed-id answer. It performs no authorization of its own — deliberately, so that a partner-created tenant can later be auto-assigned to its creator through the same function. Authorization belongs to the caller. - The partner is chosen two ways: a select of known partners, and an assign-by-email fallback
for someone granted access directly onto an existing account, who leaves no invitation behind
and so never appears in the roster. Both resolve to a Clerk user whose marker is re-read live
and must still say
audience: "partner"before anything is written. - The roster has one implementation —
readConsoleRosterinapps/console/lib/console-roster.ts, shared with/access. Clerk’s user API cannot filter bypublicMetadata, so there is no “list every partner” query; both surfaces derive the same answer from the console’s invitation list.
Tenant user management
From /tenants/:id, both audiences can invite a user, revoke a pending invitation, remove a member,
and change a member’s role — Sustentus staff on any tenant, a partner on the tenants assigned to
them. The server actions live in apps/console/app/(console)/tenants/[tenantId]/actions.ts.
Every operation writes to the tenant’s Clerk organisation, which is authoritative for who
belongs and in which role — the same record the roster reads. The platform’s existing
organizationMembership.created|updated|deleted webhook mirrors each change onto the Mongo user
collection; the console adds no sync path and writes no user document itself. A change made here
is what the tenant’s own platform sees on its next request.
The assignable roles are the platform’s. CONSOLE_ASSIGNABLE_ROLES in
apps/console/lib/console-roles.ts derives from USER_ROLES in @sustentus/services/shared minus
sales, which is a Mongo-only system role and is never offered here. Roles are written in Clerk’s
org:<role> form. Granting or revoking tenant administration is assigning or unassigning admin;
there is no separate flag or claim.
Pending invitations are listed separately — email address, role and invited date, each revocable — because a Clerk organisation invite is pending until accepted and would otherwise be fire-and-forget.
The last-admin invariant. No action of this app leaves a tenant with no admin-role member:
the last admin can be neither removed nor moved off admin. The count comes from a role-filtered
Clerk membership query rather than the 100-row roster page, and the guard runs before any write, so
a refused action leaves Clerk and the Mongo mirror both intact. It bounds this app’s actions only —
the platform’s own in-tenant user management is the second door and carries its own guard. The
check is not transactional: two operators removing two different admins from a two-admin tenant at
the same moment can both pass it, and Clerk offers no lock to close that window.
Scoping and authorisation are unchanged from the directory. Every action calls requireViewer()
and then re-resolves the tenant through tenantService.findForConsole(scopeOf(viewer), tenantId),
so a partner acting on a tenant that is not theirs gets the same not-found as for one that does not
exist. Outcomes travel back as fixed codes in the query string, never as echoed input. Removal
removes the organisation membership only — the person keeps their Clerk account and every other
tenant they belong to.
Reverting the code does not revert the writes. There is no migration, so the change is clean to roll back, but invitations already sent stay live, removed memberships stay removed and changed roles stay changed. Undoing those means the Clerk dashboard.
Plan and subscription
/tenants/:id carries a subscription card for both audiences: the tenant’s plan name, the seats it
is using against the seats it is allowed, and its subscription status. It is read-only — there is
no control to move a tenant between plans (see the limitation at the end of this section).
Clerk Billing is the system of record, read live on every request through
clerkClient().billing.getOrganizationBillingSubscription() in
apps/console/lib/console-billing.ts. Nothing about a subscription is mirrored into Mongo, and
nothing is cached: a commercial state is exactly the sort of fact that is wrong the moment it is
stored.
Seats come from the organisation, not the subscription. Clerk Billing carries no seat quantity,
so “used” is the totalCount of a one-row membership read and “allowed” is the organisation’s
maxAllowedMemberships. Counting the member page the view renders would cap every large tenant at
that page size and show it comfortably inside an allowance it had blown through. Clerk writes “no
cap” as 0, which the card states in words rather than printing a zero that means the opposite.
No money reaches the card, for either audience. TenantSubscription carries a plan name and a
status and nothing else; fee, annualFee, amount, nextPayment and lifetimePaid are in
scope only inside console-billing.ts and are never mapped out of it. The partner money boundary
therefore holds by construction rather than by a per-audience branch in the view — there is no
amount in the shape to leak.
Three answers, not two. A read returns subscribed, none (Clerk’s 404 — this tenant has no
subscription) or unavailable (Clerk’s 403 — this instance may not read Billing). Collapsing the
last two would report a paying tenant as unsubscribed the day a billing permission is revoked, so
isClerkForbidden in apps/console/lib/clerk-errors.ts is deliberately kept apart from
isClerkNotFound. Any other Clerk failure is a genuine fault and throws.
Billing is not switched on for this Clerk instance. It is a public beta, and until it is
enabled every tenant reads Unavailable for plan and status. Seats are live and correct regardless,
because they come from the organisation rather than from Billing.
Moving a tenant between plans is not possible here, and not merely unbuilt. Clerk’s backend
Billing API exposes no plan-change write — its only writes are cancelSubscriptionItem and
extendSubscriptionItemFreeTrial. A plan switch happens in the payer’s own checkout, which console
users cannot reach: they are org-less by design and never enter a tenant organisation. Offering the
move needs a writable subscription store, which is its own piece of work.
Platform markup
commercial.platformMarkupPercent is Sustentus’s platform markup on a tenant’s services revenue —
a percentage from 0 to 100, decimals allowed. It is one of the registered settings on the tenant
detail page’s Settings card, and it is set only from here, by staff: the registry marks it
consoleOnly, so the tenant’s own generic setting writes in apps/web (create, update,
remove on tenantSettingService) refuse it — creating it, renaming a row into or out of it, and
deleting its row — and the onboarding blueprint cannot propose it. The one write path is
setRegisteredSettings, which also enforces the registry’s bounds: a value below 0 or above 100 is
refused as invalid and nothing is written. Each change lands on the activity trail as
tenant.setting_changed with the key, the new value, and who made it.
Unset means no fee. The key has no registry default and is never seeded; clearing the field removes the row. There is no hidden default percentage anywhere — a reader that finds no row shows no fee rather than a £0 one. Today the markup shows on the SDM’s service-revenue figures, as a platform fee on paid revenue. It is not a go-live readiness fact: the commercial check reads the tenant’s pricing-rules flag alone, because an unset markup is a valid state the tenant’s admin cannot change.
Invoice tax rate
commercial.taxRatePercent is the tax rate on every invoice the tenant raises — a percentage from 0
to 100, decimals allowed. It sits beside the platform markup on the same Settings card and follows
the same rules: the registry marks it consoleOnly, so only staff set it here and the tenant’s own
setting writes refuse it; setRegisteredSettings enforces the bounds and records each change on
the activity trail. Unset means no tax — the key has no default and is never seeded. The rate in
force when an invoice is raised is stamped on that invoice, so a change here never restates one
already raised. What the rate does to an invoice is described in
invoicing.
Prompt workspace
Staff-only, at /prompts. The console is the edit surface for the runtime prompts the agentic
assistant answers on — the store, the versioning and the assembly are
@sustentus/services; this app is where a person touches them.
The index lists every catalogued file grouped by its ICM layer — L0 identity, L2 register
contracts, L2 rail copy, L2 surface copy, L3 reference material — with whether each is
running on a published version or on the body the repository ships, and whether an unpublished
draft is waiting. Two of those groups are composed into no prompt at all, so editing them changes a
surface the user reads rather than the instructions the model answers on: the rail files
(L2/rails/<register>.md) are what the agentic app’s
register rail says — its verb, its one-line
stance and the prose of any standing note — and the surface files
(L2/surfaces/<register>.md) are what its
composer and answer headers
say: the name answers are signed with, the mode note beside it, the chips above the input, the
placeholder and the footer line. Their file screen says so rather than reporting them as a file no
register composes yet. An opening instruction (L2/openings/<register>.md) sits between the
two kinds: it reaches the model, but on one turn only — the briefing a register gives unprompted
when a conversation starts (The register speaks first) —
and its file screen says that too. Every figure the rail shows comes from a typed server-side read instead,
which is what stops a publish relabelling a number. Beside them, and marked
read-only, sit the two code-owned halves of the same picture: the rails preamble composed into
every assembled prompt, and the L1 routing table saying which files each register reads, in
order. Showing only the editable half would misrepresent what the assistant actually runs on.
A file’s own page carries the loop: edit the markdown or upload a .md file, save it as a draft,
see a line diff against the body running right now and the fully assembled prompt the draft would
produce for each register that reads it, then publish it with a note. A draft is one per file,
overwritten by the next save and cleared by a publish, and nothing on the request path reads it
— so a draft can never reach a live agent. Publishing saves the draft first and then attempts the
publish, which is what leaves a refused body where its author can keep working on it; the
validator’s reasons render from that stored draft rather than being carried back in the URL.
Publishing and rolling back both carry the published version the screen was showing, and are refused if that pointer has moved since — someone else having published or rolled back the same file while the screen was open. The refused editor keeps their draft and is shown the version that landed against the one they tried to publish. It is a distinct refusal from the validator’s: nothing is wrong with the body, only with the world it was measured against. And it is detection, not a lock — the file is never held while someone edits it, so the window is narrowed to the moment of the write rather than closed.
When the store cannot say what is live, neither screen fills itself in from the shipped seeds:
seed text presented as the published body is how someone edits and republishes over a live version
they never saw. A store that cannot be reached at all replaces the file list and the file screen
with a notice saying so, that nothing has changed, and to try again in a moment. A file recorded as
published at a version no longer in the store is the opposite case — the store is reachable and can
name the problem — so that file is shown and flagged, with its history and its rollback control
intact. Publishing it or rolling it back both repair it: either writes a pointer that resolves, and
a file is no longer refused a publish for its own dangling pointer. What stays refused is a publish
of a different file whose assembly still composes the dangling one, because that assembly cannot
be built until the pointer is repaired — so one dangling pointer in a shared file like
L0/identity.md, composed into every register, no longer makes every file unpublishable. Publishing
is refused while either condition holds and says which it was: retrying is the next step for an
unreachable store, and repairing the named file for a dangling pointer.
History shows published versions newest first with their author, time and note, one page of 50 at
a time — promptversions is append-only, so the read is bounded in the query
(listVersions runs through findPaginated, default page size 50, served by the
{ filePath: 1, createdAt: -1 } index) rather than by the data. The card’s caption says which
versions are on screen (“Showing versions 1–50 of 123”) and a pager reaches the rest; a history that
fits on one page is captioned as showing every version. The live version is always named, even
when a rollback has put it on another page. Any version can be compared with the live one or
restored — rollback resolves its target by id (readVersion), so a version on any page rolls back
the same way. Restoring writes no new version — it moves the file’s published pointer — so the
history is never rewritten; what records who restored what is the prompt.rolled_back entry on
the activity trail below.
Staff-only three times over, and independently: the nav entry is staffOnly, proxy.ts matches
/prompts(.*) and /api/prompts(.*) in isStaffRoute, and each of the six server actions opens
with requireStaff() — the test conversation’s route handler makes the same live read through
readStaffViewer() and answers 403, because its caller is a fetch and cannot follow a redirect.
The nav is presentation — the other two are the gate. A path that is not in the code-owned
catalogue is a 404, checked before any store read.
Every interaction on the file and register screens is a plain form against a server action. The one client component on this surface is the test conversation below, which streams.
One register over time
/prompts/registers/:key is the same workspace read the other way round: not one file, but one
register’s whole prompt as it has changed. The key is the agent registry’s (assistant,
csm, vendor, expert, sdm) — the value a run record stores in agent — and its
prompt-workspace routing is looked up from it, so a register with no runs to show is a 404 rather
than an empty page.
Three things, because they are one question:
- The publish timeline — every published version of every file the register composes, newest first, with who published it, when, and their note, and a diff of any publish against the version published before it.
- Run quality either side of each publish, and it is matched by version stamp, not by
clock: every run record carries the versions that composed the prompt it answered on, so
“before” is the runs stamped with the previous version and “after” is the runs stamped with the
one this publish introduced. Counted for this register only —
L0/identity.mdis composed by every register, so a publish to it would otherwise pool all of their turns into one register’s page. Where a side has no runs the page says so; a publish nobody has answered on yet and a publish nothing has failed on are opposite facts, and 0% for both is what would let somebody sign off a bad edit. - Error and refusal rates by month, with the models that served those runs — so a rate that moved can be read against the publishes above it, against a change of model, or attributed to neither.
The pairing is “the version published before”, not “the version this replaced”. Restoring moves the file’s pointer and writes no version, so after a restore the two differ. The screen says which it means rather than claiming more than the data supports.
Scored evals
The same page runs a code-owned eval suite (packages/services/src/ai/evals/) against either the
published version set or the set the currently saved drafts would produce — which is how a draft
is scored before publishing. Four checks per case: claims present and well-formed (through
isRenderableAnswerPart, the guard the chat surface itself applies), tool scope correct (the tool
is one the register mounts, the prompt still names it, and the scope is one the question permits),
register verb honoured, and caveats present.
Every register has a case file, and each carries a <register>-house-style case: it fails when
a publish of the shared answering style (L3/how-to-answer.md) drops the rules on British
English, exclamation marks or closing offers, so a style regression shows on every register at
once.
Every grader is a string or shape comparison — no model call, no network, no clock — so the same
version set always scores the same and a failure explains itself. Each run is recorded in
agentevalruns against the version set it scored, with who ran it and whether it scored the
published or the draft-applied set.
Informative, never blocking. promptWorkspaceService.publish reads no eval result; scoring
publishes nothing and refuses nothing. Gating is a policy change in publish once the checks have
earned trust, deliberately left as a line to add rather than pre-wired as one to remove.
Talk to it
The checks say what a prompt says; /prompts/registers/:key/test shows how the assistant
behaves on it, before anything is published. It is a real, streamed, multi-turn conversation on
one register’s prompt from one of the two sources the evals score — draft, the prompt the
register would answer on if every saved draft were published, or published, exactly what it
answers on now — assembled by the same promptWorkspaceService.assembleForSource the eval action
uses, so what is scored and what is talked to never differ. The page defaults to the drafts when
any file in the register has one, and says which; it is reached from the register page beside the
scores and from a file screen’s draft card.
The page states what the conversation runs on and the route guarantees it. It is bound to the
demo tenant: POST /api/prompts/test resolves the tenant flagged isDemo on the server,
refuses unless exactly one is flagged, asserts it again with requireDemoTenant before any tool is
built, and closes every tool over it with createTenantContext — the request carries no tenant
field, and a tenantId planted in the body changes nothing. It honours the assistant switch
like every other assistant surface, refusing with the same 503 and the same sentence. The tools
are the register’s on a console caller’s terms — no persona actor and no permissions, so the
sets that read as a CSM, vendor, expert or SDM mount nothing, and the page lists each set as
mounted or not, computed by building them exactly as the route does. Nothing is persisted: no
thread, message, run record, eval record or activity entry. The client keeps the transcript and
posts it whole each turn; the route accepts user and assistant turns only (a system message is
refused, as the live route refuses it), keeps only their text parts, caps the transcript at 40
messages and 60,000 characters, and composes the system prompt on the server every turn. A
draft turn that would have applied no draft — the draft discarded while the page sat open — is
refused rather than answered on the live prompt under the wrong label.
Two consequences are deliberate. Test turns are unmetered: with no run record they never reach
/ai-spend or /agent-runs, and the gateway budget plus the transcript cap are the bounds. And
the live chat route in the agentic app is untouched — the draft-reading assembly has exactly two
callers, the eval service and this route, and a unit test pins that assembleForRegister resolves
no draft even when every file has one.
Activity trail
Every administrative action this app performs is recorded, once, in an append-only trail:
ConsoleActivity in packages/services/src/db/models/console-activity.ts, written and read through
consoleActivityService from @sustentus/services/server.
An entry carries the actor’s Clerk user id, their name and email, the audience they acted as, the
action, the subject tenant and its name, a one-line detail, and the time. Names are snapshotted at
write time, not joined. Console people are org-less Clerk users with no row in the platform user
collection, so there is nothing to populate from, and resolving each actor against Clerk would be one
API call per rendered row. Snapshotting also keeps an entry readable after the person’s access has
been revoked — which is exactly when someone comes looking at it.
What is recorded
The thirty-three recorded actions, and nothing else:
| Action file | Recorded as |
|---|---|
tenants/new/actions.ts | tenant.created |
tenants/[tenantId]/actions.ts | tenant.partner_assigned, tenant.partner_cleared, tenant.user_invited, tenant.invitation_revoked, tenant.user_removed, tenant.user_role_changed, tenant.suspended, tenant.reactivated, tenant.deleted, tenant.profile_changed, tenant.setting_changed, tenant.sla_changed, tenant.sla_removed, tenant.role_template_edited, tenant.role_template_reset, tenant.user_permissions_edited, tenant.catalogue_created, tenant.catalogue_changed, tenant.catalogue_archived, tenant.catalogue_linked, tenant.catalogue_unlinked, tenant.assistant_disabled, tenant.assistant_enabled — the configuration edits once per key, stage, role template, user or catalogue entry that actually moved, so a save that changed nothing records nothing |
tenants/deleted/actions.ts | tenant.recovered, tenant.removed |
access/actions.ts | console.access_granted, console.invitation_revoked, console.access_revoked |
prompts/actions.ts | prompt.published, prompt.rolled_back — its other three actions (saving, uploading and discarding a draft) change nothing the agents run on and are recorded nowhere |
assistant-switch/actions.ts | assistant.disabled, assistant.enabled — the platform-wide assistant kill switch, carrying the reason the flipper gave as the entry’s detail |
Recording happens after the action’s own write succeeds, so a refused or invalid action leaves no trace and a completed one leaves exactly one entry. The write is awaited and its failure surfaces: the primary write has already committed by then, so swallowing the error would mean “done and not recorded”, with nobody told.
The three access/actions.ts entries carry no subject tenant — granting someone console access
is an act against the console, not against a tenant. That makes them staff-only by construction: a
partner’s read is bounded by their assigned tenant ids, and no tenant matches none. The two
prompts/actions.ts entries are tenant-less for the same structural reason and gain the same
property: the prompt workspace is platform-wide and holds no tenant data at all. The two
assistant-switch/actions.ts entries are tenant-less for the third time over: the switch they
record is the platform-wide one, and turning the assistant off for every tenant at once is
Sustentus’s act rather than any tenant’s. The per-tenant half of that switch lives on
tenants/[tenantId]/actions.ts above and does carry its tenant.
Who sees what
/activity lists entries newest first on the directory’s own URL contract (page, 1-based; junk
reads as page 1). Staff see every entry, the tenant-less ones included. A partner sees only entries
whose tenant is currently assigned to them, resolved per request from the same assignment the
directory reads — so a re-assigned tenant takes its history out of the old partner’s view and into
the new partner’s. Denormalising partnerId onto the entry would freeze it at write time and
disagree with the directory the moment an assignment moved.
A partner is also bounded by action. They see an entry only when the console shows them the
same fact somewhere else: the tenant’s lifecycle (tenant.created, suspended, reactivated,
deleted, recovered) and its user management (tenant.user_invited, invitation_revoked,
user_removed, user_role_changed) — nine actions. Every act behind a staff-only card on tenant
detail is hidden from them: the subscription tier, the per-tenant assistant switch, the profile
and settings, SLA timers, role templates and permission overrides, the catalogue, and who the
assigned partner is. Those entries’ detail restates the fact, and for the tier and the
assistant switch carries the staff member’s free-text reason. The rule is one total map,
CONSOLE_PARTNER_VISIBLE_ACTIONS in the trail’s service, so a new action does not compile until
somebody decides whether a partner reads it. Existing entries are filtered at read time; nothing
is migrated.
The bound is applied inside consoleActivityService.listForConsole, not by the shell’s nav, so a
partner requesting /activity directly gets the same scoped answer as one who followed the link.
Staff reads declare crossTenant: true — the tenant plugin’s documented escape hatch is how a
deliberate cross-tenant read announces itself. A partner’s { tenantId: { $in: [...] } } filter
needs no hatch, because the plugin can see it. Its action: { $in: [...] } clause rides beside the
tenant clause as a residual filter on the tenantId_createdAt index.
That scoping is not restated anywhere. consoleActivityScopeFilter and consoleActivityReadModel
are exported from the trail’s service and reused by the
Notification centre, which reads these same rows through these same bounds.
The centre sets its own action clause, which would replace the partner’s, so it narrows a
partner’s notifiable set to the partner-visible one itself (notifiableConsoleActionsFor).
Immutability
Nothing edits or deletes an entry. The service exports a writer and two readers and no mutator, no
route or action offers one, and the model carries no softDeletePlugin — an audit is never edited
or removed. A unit test asserts the exported surface holds no update
or delete method.
The one thing that removes entries is permanent removal, which empties every tenant-scoped
collection and the trail is one of them: afterwards nothing tenant-scoped about that tenant remains —
its creation, its suspensions, its user changes, all gone with it. One entry survives. The
tenant.removed entry is written tenant-less, snapshotting the removed tenant’s name, so the
console still records that Sustentus removed a workspace and who did it; being tenant-less it is
staff-only, which is the right audience for a tenant that no longer exists.
Indexes
autoIndex is off on this schema, so both indexes come from migrations rather than from model
registration:
| Index | Serves |
|---|---|
createdAt | the staff feed — every entry, newest first, across every tenant |
tenantId_createdAt | a partner’s $in feed, and any future per-tenant read |
createdAt deliberately does not lead with tenantId, unlike the rest of this database: the staff
read of this collection has no tenant clause to lead with, and a single-field createdAt is not a
prefix of the compound one. Both are created with the schema’s { locale: "en", strength: 2 }
collation, named explicitly — an index created without a collation inherits the collection’s
default, and autoCreate (a separate switch, still on) stamps the schema’s collation onto the
collection, so leaving it unnamed would make the result depend on whether the migration or the app’s
first model compile got there first.
Notification centre
The trail records; the centre tells. /activity and the dashboard feed are both places somebody
has to go — a partner learns a tenant of theirs was suspended by opening the console and looking.
The notification centre is the same information arriving instead: a bell in the shell header
carrying an unread count, a popover previewing the newest unread, and /notifications as the full
list.
A projection of the trail, not a second record
There is no notification collection and no fan-out write. A notification is a
console-activity entry, seen by somebody who is in scope for it. The centre —
consoleNotificationService in
packages/services/src/db/services/console-notification/ — reads that one collection with three
extra predicates over the trail’s own scoped filter, and lays per-person read state on top:
| Predicate | Why |
|---|---|
action in the notifiable set | seven of the trail’s twenty-four actions |
| the trail’s own audience scope | identical to what /activity shows |
actorClerkUserId not the viewer | nobody is told of their own action |
Three properties follow from that and are not maintained by anything:
- The two views cannot disagree. Nothing writes a notification, so nothing can fail to write one, and no reconciliation job is needed to keep a second collection in step with the first.
- Removal is inherited. Permanently removing a tenant empties its trail entries, so its
notifications go with them. The one tenant-less
tenant.removedentry the trail keeps back survives as a staff notification naming the removed workspace — same row, same rule. - Marking read touches the trail not at all. Read state is a separate per-person collection, so the audit stays append-only and the same entry can be read by one person and unread by another.
Which entries are notifiable
Seven of the twenty-nine recorded actions — the ones about a workspace existing, its lifecycle moving, or somebody’s power over it changing:
tenant.created · tenant.suspended · tenant.reactivated · tenant.deleted ·
tenant.recovered · tenant.removed · tenant.user_role_changed
The set is CONSOLE_NOTIFIABLE_ACTIONS, declared as a total Record over
ConsoleActivityAction in the shape CONSOLE_ACTIVITY_LABELS already uses. A new trail action
therefore stops the file compiling until somebody decides whether it is notifiable — the twenty-two
that are not (partner assignment and clearing, the three user-management actions, the three
console.* access actions, the two prompt.* ones, the seven tenant-configuration ones and the
five tenant.catalogue_* ones) are false there rather than absent.
tenant.profile_changed, tenant.setting_changed, tenant.sla_changed, tenant.sla_removed,
tenant.role_template_edited, tenant.role_template_reset, tenant.user_permissions_edited and
the five tenant.catalogue_* actions are silent by decision, not omission: the console trail is
the whole audit of a staff configuration edit — role templates, permission overrides and catalogue
entries included — and the tenant is deliberately told nothing about one. The staff actor is captured on the tenant’s own audit arrays (RoleTemplate /
UserPermissionOverride) as a snapshot alongside the app-user by reference, so the tenant’s
in-app audit views render the staff member’s name without the console ever creating a platform
user row for one of its own people.
There is no plan-change notification because there is no plan-change action to project — see
Plan and subscription above. When plan movement becomes possible it is
one more trail action and one more true in that map.
Who is told
Scope is the trail’s scope, resolved live per request, and it is the trail’s own code rather
than a copy of it: consoleActivityScopeFilter and consoleActivityReadModel are exported from
db/services/console-activity/ and used by the centre. A partner’s
{ tenantId: { $in: [...] } } bound is the whole of partner authorization on this collection, so
a second copy would be a second place for it to widen.
Both of those functions match scope.audience === "sustentus" positively rather than testing for
!== "partner". The union has two members, so the forms are equivalent today — but only the staff
audience can then reach the unbounded filter and the crossTenant escape hatch, and a third
audience added later is bounded by default instead of inheriting staff’s view.
actorClerkUserId: { $ne: <viewer> } is applied in the query, not as a filter over the
results, so the list, the unread count and the page count all agree about which rows exist.
Read state
Two collections in packages/services/src/db/models/console-notification.ts, neither carrying a
tenantId — so neither takes tenantPlugin, neither needs a demo-reset policy classification
(policy.test.ts classifies schemas that carry tenantId), and neither is reached by
tenant-purge:
| Collection | Shape | Written by |
|---|---|---|
console-notification-cursor | one per person: clerkUserId, allReadBefore | Mark all read |
console-notification-read | one per person per entry: clerkUserId, activityId | Mark read |
A notification is read when its entry predates that person’s cursor or its id has a read row.
Mark all read is O(1) — stamp the cursor, then delete that person’s read rows, all of which name
entries the new watermark now subsumes. The read rows are bounded because markRead performs the
same compaction the moment nothing is left unread: without that, somebody who only ever marked rows
individually would never reach the operation that prunes them.
Both writes are upserts against a unique index, which is not atomic — two submissions of the
same mark race, and the loser’s insert raises a duplicate key. Both catch it through the house
isDuplicateKeyError, because the operation had in fact succeeded.
markRead re-resolves the entry through the same scoped, notifiable, not-your-own read before
writing anything, so a read row can only ever exist for a notification that was genuinely the
caller’s. Server actions are dispatched by action id and can be posted from any path; that
re-resolution is the boundary, not the page that renders the button.
Indexes come from a migration with autoIndex off, following the console-activity precedent, and
name the schema’s { locale: "en", strength: 2 } collation explicitly for the same reason the trail’s
do. No index was added to console-activity: staff sort on createdAt and a partner matches
tenantId then sorts, which the trail’s two existing indexes already serve; the notifiable action,
the actor exclusion and the read-row exclusion are residual filters over a set those have already
bounded, at a volume of a handful of administrative actions a day.
Where a notification goes
Derived from the action alone, with no extra lookup per row (notificationHref in
apps/console/lib/console-notification-links.ts):
| Action | Destination |
|---|---|
tenant.removed | none — the entry carries no tenant and the tenant is gone |
tenant.deleted | /tenants/deleted for staff; unlinked for a partner, who has no deleted-tenant surface |
| everything else | /tenants/:id, itself scoped, so a link a reader should not be able to follow behaves exactly as a typed URL does |
That file imports only a type, so it holds no server dependency and is unit-tested directly.
What the centre does not do
- It publishes nothing realtime. No Ably message —
packages/services/src/notifications/is the tenant platform’s per-resource notify path and is not involved here. The centre itself is an in-app projection for people signed into the console. The same events are now delivered by email, by a separate path that reads this same set — see Email notification below. - Marking read never removes a notification. The row stays, shown as read, because the trail entry behind it is the audit.
- Opening one does not mark it. Reading is an explicit act.
- Nothing is realtime. The count and the preview are read in the shell’s server component when
the page renders. Both mark actions
revalidatePath("/", "layout"), so the count is right after either — but App Router does not re-render a shared layout on client-side navigation between the routes under it, so a count is otherwise as of the last full load. Moving the header into a(console)/template.tsxis what would change that. - The mark-all-read watermark is global, while scope is resolved live. A tenant assigned to a partner after they last marked all read arrives with its history already behind their cursor, so those entries are in their centre — the list is correct — but never raise the badge. Fixing that means a per-scope watermark.
Email notification
The centre tells whoever signs in. Email tells the people who will not — a tenant’s own administrators, who have no console account at all, and an operator who is not looking today.
The event set is the centre’s, not a second list. CONSOLE_EMAIL_MATRIX
(packages/services/src/notifications/console/audience.ts) is a total Record over
ConsoleActivityAction saying which of three audiences each action reaches, and a unit test
asserts both directions against CONSOLE_NOTIFIABLE_ACTIONS: every notifiable action emails
somebody, and nothing the centre stays silent about sends mail. The two channels cannot drift.
| Action | Tenant admins | Partner | Staff |
|---|---|---|---|
tenant.created tenant.suspended tenant.reactivated tenant.deleted tenant.recovered | yes | yes | yes |
tenant.removed | yes | — | yes |
tenant.user_role_changed | — | yes | yes |
| the twenty-two non-notifiable actions | — | — | — |
tenant.removed reaches no partner because the entry it projects carries no tenant — the same
reason it is absent from a partner’s centre. Its tenant-admin recipients are read before the
Clerk organisation is deleted and handed to the send, because afterwards there is no membership
left to read.
Two vocabularies
A tenant administrator is not a console user, and their mail says so: no console link, no console
words, no actor name — they are told what happened to their workspace, with
support@sustentus.com as the contact and the Reply-To. Partner and staff mail is operator-voice
and links back to the tenant at the same destination notificationHref gives the centre’s row, so
an email and a notification about one event land on the same page. With CONSOLE_APP_URL unset the
operator mail sends linkless rather than carrying a link to nowhere.
Nobody is told about their own action — matched on the actor’s Clerk id or their address,
because a recipient can reach the fan-out without an id (SYSTEM_ADMIN_EMAIL is a bare address, and
so is a pending invitation).
Recipients
Resolved in apps/console/lib/console-event-email.ts, because every one of them is a Clerk read —
which is this app’s dependency, not the services package’s:
| Audience | Resolved from |
|---|---|
| Tenant admins | org:admin memberships and pending org:admin invitations on the organisation |
| Partner | the tenant’s assigned partnerId, or nobody when it has none |
| Staff | the console invitation roster, plus SYSTEM_ADMIN_EMAIL when it is set |
The invitations half is not a nicety: a new tenant’s organisation is created empty — console people
are org-less, so there is no createdBy to make a first member — and its first administrator exists
only as an invitation. Reading memberships alone would find nobody at precisely the moment the
“your workspace is ready” mail is meant to send.
The roster half is likewise: it is derived from invitations this console created, so it cannot see somebody granted access directly on an existing account — the bootstrapped admin especially. The env address closes that without a second mechanism, and the fan-out deduplicates by address, so naming somebody who is also on the roster costs nothing.
Failure is recorded, never raised
The send runs in after(), after both the administrative write and the trail entry have committed,
and nothing in it can affect the action that caused it: every path resolves, the after() call
is itself wrapped, and a total failure is a set of rows rather than an error an operator sees. A
suspension that succeeded stays succeeded, and the fact that nobody was told is written down instead
of lost.
console-email-delivery-log is that record — append-only, failures only, bounded by a 30-day
TTL index created by migration with autoIndex off. It is its own collection rather than a row in
the platform’s notification-delivery-log, whose tenantId is required by tenantPlugin and
which the tenant-less tenant.removed cannot supply, and whose recipients are platform user rows
console people do not have. That separation is about what is being recorded, not about reach:
the console also reads the platform’s log, cross-tenant, through
consoleNotificationDeliveryLogService — see Fleet delivery below.
| Reason | Means |
|---|---|
not-configured | RESEND_API_KEY is unset in this environment |
transport-error | the provider rejected or threw |
recipient-missing | a resolved recipient carried no address |
recipients-unresolvable | the Clerk read for that audience failed |
Deliberate non-delivery is never a failure. A tenant with no partner, or an audience the actor was filtered out of, produces no row — an empty audience is a fact, an unreadable one is a fault, and only the second is logged.
Staff read the failures at /notifications/delivery — gated twice, by the proxy’s staff matcher
and by requireStaff() on the page — reachable from a staff-only nav entry so it can be opened to
confirm health, not only to diagnose breakage. The dashboard reports the trailing 24 hours in both
states — an alert when sends have failed, distinguishing a missing provider key from per-send
faults, and an all-clear when none have. Both reads declare
crossTenant: true: the question is whether console mail is getting out at all, which is about the
estate, and the tenant-less removal rows no tenant filter could reach.
Fleet delivery
Staff-only, at /notifications/fleet. Where the section above is the console’s own mail, this is
the platform’s notifications — the in-app rows, the Ably pushes and the Resend emails every
tenant’s people are meant to receive.
Those are fire-and-forget by design: createNotification never throws to its caller, and a dead
Ably key, an unregistered template or a missing address degrades to a warn-and-continue no-op.
notification-delivery-log already recorded each failure, but the only surface onto it was each
tenant’s own /admin/notifications page — a page tenant admins rarely open and staff cannot open
at all. A platform-wide breakage therefore surfaced as N separate tenant logs nobody was reading.
This page adds them up.
Read-only, and cross-tenant by construction. The reads live in
consoleNotificationDeliveryLogService, a console-scoped entry point beside the tenant-scoped
notificationDeliveryLogService rather than a flag on it, so the exemption never reaches the
surface tenant admins use. Nothing here writes: the only writer of that collection stays
notifications/core/record-failure.ts, and the fix for any row happens where the failure points —
a template, a key, a recipient.
The three reads declare their exemption in two ways, and the difference is the point:
| Read | Shape | How the exemption is declared |
|---|---|---|
recentFailures | the capped row list | .setOptions({ crossTenant: true }) — tenantPlugin hooks find and honours it |
failureBreakdown | one $group behind the fleet page’s summary line and filter options, tenant names resolved | named in a comment at the pipeline — tenantPlugin does not hook aggregate at all, so there is no flag to declare |
failureSummary | one $group behind the dashboard alert — total, tenants affected, misconfigured; no tenant names, no option lists | the same comment at its own pipeline; it shares the window $match and the misconfigured rule with failureBreakdown, so the two counts agree |
No new index and no migration: the unscoped window read is served by the collection’s existing
createdAt_ttl index traversed in reverse, a tenant-filtered one by tenantId_createdAt, and the
same TTL is what keeps the grouped read cheap.
What the page says out loud. It covers seven days, matching the tenant-side page so the two
never disagree, and states the log’s 30-day retention beside it — an empty page is not a clean
month. It is capped at the 50 most recent, like /agent-runs and for the same reason: a second
page of fleet failures would mean something is wrong at a scale a pager does not help with. The
summary counts and the filter options are the whole estate’s even when the list is narrowed,
because the options are built from that same read, and the page says so rather than leaving a
reader to trip over it.
Filters — tenant, notification type and channel — are applied in the read, not the render, and each is validated against the thing that makes it meaningful: a tenant by shape, so a tenant that failed nothing keeps its filter and gets the empty list that is the true answer; a channel against the model’s closed union, for the same reason; a type against the window’s own types, because the stored type is an open string where an unmatched value is far likelier a typo than a quiet notification.
The recipient is deliberately not resolved here. The tenant-side log names them because it is
already inside that tenant; doing it on a fleet view would widen a triage read into every tenant’s
user records for no diagnostic gain. Staff follow the tenant link — present on every row, and
rendering a deleted tenant unlinked under its id rather than dropping it, because those failures
were real and hiding them would stop the totals adding up. A tenant’s own slice is reachable the
other way too, from a staff-only card on /tenants/:id that reports its seven-day count and links
here pre-narrowed — that card reads the tenant-scoped service, since there a tenant leads the
query and a cross-tenant flag would misdescribe it.
Assistant health
Staff-only, at /assistant-health. The agentic chat route
(apps/agent/app/api/chat/route.ts) degrades rather than fails: when the conversation store is
unreachable the turn is still answered and no conversation is stored for it, because a database
blip should cost a user their history rather than their assistant. The run record is still
written, marked persistence: "unsaved", so the turn’s spend is metered even though its
conversation was lost. That choice left no trace beyond a
console.error, so an outage looked from the outside like a product that works. This page is the
trace.
agent-degradation-log is the record — append-only, calls that did not happen only, bounded by
a 30-day TTL index created by migration with autoIndex off, exactly as
console-email-delivery-log above. One row per store call the route swallowed:
| Field | Means |
|---|---|
operation | which call was lost — thread read/create, message writes, history read, run record, or the register rail’s read |
outcome | what it cost: served-unsaved, history-lost, write-lost, or rail-lost |
cause | failed (the call threw) or refused (the store answered and declined) |
errorCode | the error’s name only — never the driver’s message text. Present exactly when cause is failed |
firstOfTurn | true on the one row per turn that tipped it into degradation |
One operation here is not a conversation-store call. The agentic app’s register rail is read on
every page render, and a read it loses is written as rail read / rail-lost: the page still
serves the register’s verb card and its published copy, and only the figures are missing. It shares
this vocabulary rather than getting a collection of its own, so “is the assistant losing reads” is
answered from one place. It is also the one operation that never carries firstOfTurn — that field
counts degraded turns, and letting a per-render read claim it would make the headline figure
count navigations instead.
cause is a third axis, not a fourth outcome. A refused write costs the turn exactly what a
failed one costs (write-lost), so the two are orthogonal; folding them together would need a
*-refused twin of every outcome. The distinction earns its place on the deploy-to-migration
window: appendMessage matches on nextSequence: { $exists: true }, so a thread the
agent-thread-next-sequence migration has not reached is refused rather than written at a position
that was never reserved. Production migrations run behind a required reviewer, so between a deploy
and its approved migration the assistant answers normally while conversations stop recording — a
run of refusals on this page is what that looks like, and the page says so. Rows written before the
field existed carry no cause and are read as the failures they were.
firstOfTurn is the counter. A turn that loses its thread read, its history read and its run
record is one degraded turn, not three, so “how often is the assistant serving turns it cannot
save” is countDocuments({ firstOfTurn: true }) over a window rather than a row count.
Recording can never cost a turn, and costs one write. The work is handed to after() and
swallows everything, including its own failure — which matters more here than for the email log,
because it goes to the very database that is already failing. A turn’s rows are collected in memory
and written as a single insertMany after the response, so the model — and therefore connectDB(),
and therefore a server-selection timeout — is resolved once per degraded turn rather than once per
lost call. firstOfTurn is the batch’s first row, decided before anything is written rather than
raced for between writes. Do not reintroduce a per-row reporter: it turns the cheapest path in the
route into a multiplier on connection churn exactly when the pool can least absorb it. tenantPlugin
hooks queries and not insertMany, so every document sets tenantId explicitly.
The collection is the readable record; a drained log line is the durable one. The collection
lives in the database the degradation is usually about, so a total Mongo outage takes it with the
store calls it reports. recordAgentDegradations therefore mirrors every batch to the agent app’s
standard error before it resolves the model — before connectDB(), before server selection,
before any timeout — as one line of JSON whose first key is the marker sustentus.agent-degradation.
The line carries exactly what a row carries (tenant, operation, outcome, cause, the error’s name,
firstOfTurn) and is built field by field so it cannot carry anything else. A window
/assistant-health shows nothing for is reconstructed from that mirror; the page says so, always,
because a gap and an all-clear look identical.
The sink is a Vercel log drain on the agentic-interface project, and that was chosen over
PostHog and over Sentry: no vendor SDK, no API key, no dependency and no turbo.json → globalEnv
entry. What it buys is durability and only durability — no alerting, no grouping, no dedupe, no
retention policy. Nobody is paged by a mirrored line.
Filter to the marker — at the destination, not at the drain. Draining the function log means the
log leaves the server, and the console.error lines the chat route writes beside each degradation
carry the raw driver error, hostnames and credentials included. The mirror line is redacted by
construction; those lines are not, and are deliberately left that way — the raw error next to the
stack is what a developer reads in Vercel’s own log view.
A Vercel log drain has no per-line content filter: it selects sources (lambda, edge,
build, static), environments, projects and a sampling rate, and forwards whole lines. So the
filter is the destination’s job — an ingest rule or pipeline processor that keeps only lines whose
first key is sustentus.agent-degradation and drops the rest. Point the drain at a store that can
do that, and configure the rule when you configure the drain; a drain into a bucket that keeps
everything is the thing this warns against.
The reporting path’s own two failure lines were reduced to error names for this reason and need no
filtering: recordAgentDegradations’ warning (which fires one line after the mirror, on the very
outage the mirror exists for) and the agent reporter’s guard in
apps/agent/lib/degradation-reporter.ts. Only the chat route’s per-call lines are still raw.
Staff read it at /assistant-health, gated twice like the email log — the proxy’s staff matcher
and requireStaff() on the page — reachable from a staff-only nav entry so it can be opened to
confirm health, not only to diagnose breakage. Both reads declare crossTenant: true: the
question is whether the assistant is persisting at all, which is about the estate. The tenant name
on each row is resolved at read time rather than denormalised at write time, because a name lookup
on a failing store would fail with the rest of it.
Agent runs
Staff-only, at /agent-runs. /assistant-health above answers whether a turn was saved; this
page answers whether it worked. Before it, a failed or refused turn produced a console.error
on a serverless log nobody reads and a run record no screen rendered — so the first signal was a
user complaint.
Nothing new is written for it. The run record (agentruns) already carries the register, the
model, the outcome and its error code, the tool calls with their bound tenant and effective scope,
and the prompt version set behind the answer; this page reads it. Failures first and by default,
with every turn one click away for the times the question is “what does normal look like”,
filterable by register and by month.
Each row expands the same TraceRail the conversation surface uses, and lists the turn’s prompt
version set with a link per file into the prompt editor’s compare view — so the text behind a
failed answer is one click from the failure rather than a search. A file served from its
repo-shipped seed is labelled as the seed rather than linked to a version page that would 404.
The tenant shown against a tool call is the call’s own binding, never the run’s. A tool is closed over one tenant at build time, and this is the screen built to audit that — so a call bound somewhere other than the run’s tenant renders as its own id rather than under the run’s name.
Capped rather than paginated, like the assistant-health log and for the same reason: it is a diagnosis aid, and a second page of failures would mean something is wrong at a scale a pager does not help with. The population is stated on the page — including how many of the month’s runs carry no version stamp, because those predate it and belong to no version.
The reads are unscoped by design and staff-only for it: a prompt version has no tenant, so “how did
runs on this version go” is cross-tenant by construction. The one find declares
crossTenant: true; tenantPlugin does not hook aggregate at all, so the pipelines carry the
staff gate at their callers instead — proxy.ts’s staff matcher and requireStaff() on the page.
Home dashboard
/ is the console’s landing surface, scoped to the viewer’s audience:
- Total organisations — every live tenant in scope: active + suspended, exactly what the directory lists.
- Active organisations — those in the
activelifecycle state, so the gap between the two numbers is the suspended ones. Deleted tenants are in neither count, and recovering one returns it to both. - Recent activity — the newest entries from the same scoped read
/activityuses, rendered by the same component, so the two surfaces cannot describe an entry differently. - Email delivery — staff only, in both states: the alert when the trailing 24 hours contain a
failed send, and an all-clear when they do not. The all-clear reports the absence of failures
and deliberately does not claim mail is getting through: successful sends are never logged, and
both counts in
failureSummary()come from failure rows, so an environment with no provider key and no administrative actions in the window produces the same zero as a healthy one. It names its own 24-hour window because/notifications/deliverylists 30 days. The summary read is skipped entirely for a partner rather than merely hidden, so a partner never causes an estate-wide count. - Fleet delivery — staff only, built to the same idiom and for the platform’s notifications rather than the console’s own mail: both states, the same any-failure-in-24-hours threshold, the same separation of a missing provider key from per-message faults, and the same refusal to read an all-clear as proof of delivery. What makes it the fleet signal is the tenant count beside the total — one tenant failing is that tenant’s problem, the same count across a dozen is a platform breakage. Its read is skipped for a partner too.
The counts are countDocuments over the console-scope filter builders, extended with a lifecycle
selector. They leave the soft-delete clause to softDeletePlugin, which hooks countDocuments as
well as every find* and adds its live clause as an $and entry — beside the lifecycle clause, not
merged into it. The one console filter that still states the clause itself is the single-tenant id
filter, because it also feeds Tenant.updateOne and the raw collection, which the plugin never
hooks.
There is no platform-health or uptime figure — that needs a real definition before it is worth printing one.
Data
The console reads MongoDB through @sustentus/services/server, so it needs MONGODB_URI and
MONGODB_DATABASE_NAME in every Vercel scope it is deployed to. Outbound email needs
RESEND_API_KEY and EMAIL_FROM (the bare verified sender address; the display name is applied
in code), CONSOLE_APP_URL for the operator
mail’s link, and optionally SYSTEM_ADMIN_EMAIL. These belong to the tenant-management Vercel
project specifically — being set on web does nothing for the console. With the key unset the
console still works: every intended recipient is recorded not-configured and the dashboard says
so. MONGODB_URI is required and
throws when absent; MONGODB_DATABASE_NAME is optional to the driver and silently falls back to
the database named in the URI path, so it should be set explicitly rather than inferred.
Running it
pnpm --filter @sustentus/console dev # port 3006