Skip to Content

← All archived runs

Run: ai-gateway-billing-unlock

run.md

Run: ai-gateway-billing-unlock

  • branch: claude/ai-gateway-billing-unlock-pipeline-2ae3ax
  • pr: #900

02_define/output/spec.md

Spec: Unlock AI Gateway billing and make AI failures visible

  • slug: ai-gateway-billing-unlock
  • personas: Admin, Vendor, Customer
  • touches: apps/agent, apps/web, apps/docs
  • complexity: standard

Problem

Every agent on the platform is addressed through the Vercel AI Gateway, and the gateway has no credits — the free tier 403s on anthropic/claude-haiku-4.5. Live AI is effectively down: the two shipped agents (BRD, onboarding concierge) cannot be exercised end-to-end, the agentic app's new conversation cannot be smoked against a real model, and the three roadmap agents behind this batch are all blocked on the same wall. Jamie's note: "AI agents are not testable as the billing mechanism is locked." Until it is unlocked, nothing in the Scale the bridge initiative's AI half can be built or believed, and Q2-2026 Objective 2 — Build Repeatable Lead Generation Pipeline — depends on the BRD and matching agents actually running.

The outage was also invisible in the wrong direction. When the gateway declined, the raw provider error reached the tenant's screen — provider name, account tier and a Vercel billing top-up link — and hard-locked fresh tenants on onboarding until the fix-onboarding-skip lane (PR #791) added a skip. That lane fixed the onboarding concierge's copy and agentic-chat-interface shipped friendly decline copy on the agentic conversation, but the BRD chat still renders error.message verbatim to a customer, and nothing anywhere tells a user the assistant is unavailable before they have typed a message and lost the turn.

Proposed change

Three halves, one ops and two in-product.

Ops (done, still to be documented). The gateway is funded as pay-as-you-go credits with a $100/month spend alert — confirmed by Jamie on 2026-08-27, so the wall this batch is named after is already down. What remains is the runbook that makes the arrangement legible instead of tribal: who owns the Vercel account, where the balance lives, the top-up path, the alert threshold, and how to smoke-test live AI without spending a demo. Funding is an account action in the Vercel dashboard, not code — this run records it, it does not perform it.

Availability signal (apps/agent). A server-side probe reads the gateway credit balance and reports a coarse verdict to the conversation, so a credit outage is visible before a user hits it mid-turn. The verdict is coarse on purpose: the balance figure, the account tier and any gateway error text are account-internal and must never reach a signed-in tenant user — the same class of leak this run is closing. When the gateway cannot serve, the conversation shows a non-blocking notice above the composer; the user may still send, because a probe is evidence and not a verdict on the next request.

Copy-only fallback (apps/web). The BRD qualification chat renders fixed copy instead of the raw error. This is defensive text on a surface that is live to customers today, not a feature — it is deleted, not extended, when the platform's AI surfaces become redirections into the agentic app. The onboarding concierge and the agentic conversation already carry their fallback copy and are left alone.

Acceptance criteria

  • GET /api/ai/availability in apps/agent answers a signed-in caller with a coarse verdict — ok, degraded, or unknown — derived from the gateway credit balance, and refuses a signed-out caller with the same deny-by-default gate the chat route carries.
  • That response never contains the credit balance, the account tier, a provider or gateway error message, or a billing URL. The detail goes to the server log only.
  • The probe result is cached server-side, so opening the conversation repeatedly does not issue a gateway call per mount.
  • When the verdict is degraded, the agentic conversation shows a non-blocking notice before the user sends anything, and sending is still permitted.
  • When the verdict is unknown — the probe itself failed, or no gateway credential is configured — no notice renders. A check that cannot run must not tell every user the assistant is down.
  • The BRD qualification chat (apps/web/components/projects/brd-chat/brd-qualification-chat.tsx) renders fixed copy on failure; no provider name, account tier, gateway text or billing URL reaches the DOM, and the raw error is logged to the browser console instead.
  • A billing runbook page exists in apps/docs under technical/ naming David as the Vercel account owner, the pay-as-you-go credits arrangement, where the balance is read, the top-up path, the $100/month spend alert threshold, and the procedure for smoke-testing live AI without running a canned demo.
  • A live (non-canned) BRD chat and onboarding chat each complete a turn successfully in preview and in production.

Out of scope

  • Funding the account. Buying credits and setting the alert is an account action in the Vercel dashboard, not code. Jamie confirmed on 2026-08-27 that this is done: pay-as-you-go credits, alert at $100/month. The runbook records the arrangement; no code in this run performs, reads or verifies the billing setup.
  • A spend cap. Jamie's ruling (Q2, 2026-08-17): no cap in v1 — alert threshold only, and product behaviour never changes as spend rises. The availability signal reports cannot serve, never spending a lot.
  • A staff-facing availability dashboard. Raised at pickup as the stub asked and ruled out: the balance alert is the operator's signal. If one is ever wanted it belongs in apps/console, never in apps/web.
  • The agentic conversation's decline copy. agentic-chat-interface already shipped it (CHAT_ERROR_MESSAGE in apps/agent/components/assistant-chat.tsx); the cross-batch question the stub flagged is settled in that stub's favour. This run does not re-own, restyle or duplicate it.
  • The onboarding concierge's error copy. Already fixed by the fix-onboarding-skip lane.
  • Per-tenant AI usage metering or billing.
  • New agents, new tools, or model changes — DEFAULT_AGENT_MODEL and DEEP_EXTRACTION_MODEL stay as they are.
  • Any new AI surface in apps/web, and any redirection of the existing ones into the agentic app.

Open questions

  • none

The one question Define opened was settled inside it. The alert threshold is $100/month (Jamie, 2026-08-27), so the runbook carries a real figure and no placeholder. It drives no behaviour: Q2 already ruled that product behaviour never responds to spend, and the availability probe reports cannot serve, never spending a lot.

Notes

No scope.md behind this batch, by design. .icm/intake/ai-platform-enablement/breakdown.md records that the batch was cut from Jamie's platform-audit notes of 2026-08-17 plus the web-app feature audit, with the notes quoted per stub, rather than from a Scope run. There is no .icm/runs/ai-platform-enablement/ and nothing for /pipeline approve to settle, so Define took the stub as the settled input. Jamie's recorded rulings on the stub's Q1 and Q2 bind this spec and are honoured above.

Funding landed during Define. The stub was written while the gateway was unfunded, and its headline criterion assumed Build would wait on an account action. Jamie funded it on 2026-08-27 and set the alert at $100/month, so the last acceptance criterion is verifiable at Verify rather than blocked on someone. This does not shrink the run: the guardrails are the half that stops a repeat, and an outage that is now impossible to hit by accident is exactly the condition under which the degraded-state path has to be tested deliberately — see the runbook's smoke-test procedure.

How the probe works, and why this shape. ai re-exports the gateway provider (import { gateway } from "ai"), whose getCredits() returns the remaining balance without spending a model call — so availability is established without burning tokens to ask. It reads AI_GATEWAY_API_KEY, already declared in turbo.jsonglobalEnv, so this run adds no environment variable. The probe lives in an apps/agent route handler rather than in packages/services: apps/agent may import only the /ai subpath, which is declared client-safe with no process.env reads (apps/agent/AGENTS.md), and a credential-reading balance call does not belong behind that contract. apps/agent/proxy.ts already matches /(api|trpc)(.*) deny-by-default, so the new route is gated by the edge on arrival; the in-route session check is the second gate the chat route carries for the same reason.

Context budget: within band. Beyond the Inputs table, this stage read the four apps/web and apps/agent source files the stub named (to establish which leaks are already fixed and which remains), apps/agent/AGENTS.md and proxy.ts (the services boundary and the route gate), and the installed @ai-sdk/gateway type surface (to confirm getCredits() exists before speccing on it).

03_build/output/notes.md

Build notes: ai-gateway-billing-unlock

  • commits: feat: ai-gateway-billing-unlock — availability signal, chat error copy, billing runbook
  • ci: GREEN on 44094a1 — every blocking check passed; the agentic-interface, docs and web previews all built, which are exactly the three apps this diff touches

What changed

  • apps/agent/lib/ai-availability.ts (new): verdictFromBalance maps a gateway credit balance to ok | degraded | unknown, and readAiAvailability wraps it in a 60s in-process cache. gateway.getCredits() establishes availability without spending a model call to find out. Both an unparseable balance and a thrown probe resolve to unknown, never degraded — our own failure must not become a claim about the product.
  • apps/agent/app/api/ai/availability/route.ts (new): GET, session-gated, returns { status } and nothing else. The balance, the account tier and gateway error text stay in the server log. Lives in the app rather than packages/services because apps/agent may import only the /ai subpath, which is declared client-safe with no process.env reads (apps/agent/AGENTS.md) — a credential-reading balance call does not belong behind that contract. proxy.ts already matches /(api|trpc)(.*), so the edge gates it on arrival; the in-route check is the second gate the chat route carries for the same reason.
  • apps/agent/components/assistant-chat.tsx: probes availability on mount and renders a non-blocking warning alert above the composer when the verdict is degraded. State starts at unknown and unknown renders nothing, so a probe that fails stays silent. Sending is never disabled — the probe is evidence about a moment ago, not a verdict on the next request.
  • apps/web/components/projects/brd-chat/brd-qualification-chat.tsx: renders fixed CHAT_ERROR_MESSAGE copy instead of {error.message}, and logs the raw error to the console. This was the last surface still putting the provider's text — provider name, account tier, billing link — in front of a customer.
  • apps/docs/app/technical/ai-gateway/page.mdx (new) + a link from the technical index: the billing runbook — owner, arrangement, where the balance lives, top-up path, the $100/month alert, what each surface does on decline, and how to smoke-test live AI (including how to provoke the degraded path deliberately).

No new environment variable: the SDK reads AI_GATEWAY_API_KEY, already declared in turbo.jsonglobalEnv. No change to packages/services — the agents and model constants are untouched.

Acceptance criteria status

  • GET /api/ai/availability answers a signed-in caller with a coarse verdict, and refuses a signed-out one — 401 in-route, plus the edge gate. Asserted in route.test.ts.
  • The response never contains the balance, tier, gateway error or billing URL — the body is { status } and the test asserts the whole object, so a leaked field fails it.
  • The probe result is cached server-side — 60s TTL; ai-availability.test.ts asserts one gateway call inside the window and a second after it.
  • degraded shows a non-blocking notice before the user sends anything, and sending is still permitted — the alert sits above the composer; the submit handler is unchanged.
  • unknown renders nothing — the component's initial state is unknown and a failed or non-ok probe leaves it there.
  • The BRD chat renders fixed copy; the raw error goes to the console. No error.message remains in the file.
  • The billing runbook exists in apps/docs under technical/, naming David, pay-as-you-go credits, the balance location, top-up path, the $100/month alert and the smoke-test procedure.
  • A live BRD and onboarding chat each complete a turn in preview and production — not assertable from code. This is Verify's DoD smoke on the preview and a production check after deploy; the runbook's smoke-test section is the procedure.

Notes for Verify

  • The last criterion is yours, not Build's. It needs a human sending one live turn on a non-demo tenant in the preview, and again on production after deploy. A passing canned demo is not evidence — canned mode never reaches the gateway.
  • Check the unknown branch deliberately. It is the one most likely to be got backwards: a probe that fails must stay silent, not warn. Now that the account is funded, neither degraded nor unknown can be reached by using the product normally — the runbook documents how to force each, and the unit tests pin the mapping.
  • The cache is per-instance, which is correct for a hint of this kind but means two serverless instances can briefly disagree. Nothing depends on them agreeing.
  • The onboarding concierge and the agentic conversation's own decline copy were already correct (fix-onboarding-skip and agentic-chat-interface respectively) and are deliberately untouched.

Context budget: within band. Beyond the Inputs table this stage read packages/ui/src/lib/variants.ts (to pick a real Alert variant rather than guess one) and apps/agent/tsconfig.json (to confirm the @/* alias the new import relies on).

04_verify/output/verify.md

Verify: ai-gateway-billing-unlock

  • ci: GREEN on 6a8541f — settled via ci-status.sh after the last code push. The verify.md commit that follows is markdown-only; see "Preview caveat" below.
  • previews smoked: none by the agent — all Vercel previews on this team sit behind Vercel SSO deployment protection. A plain fetch of the agentic-interface preview returns 302 to vercel.com/sso-api, which is Vercel's wall and not the app's gate, and the Vercel MCP could not mint a shareable URL. Every preview-demonstrated line below is therefore the operator's. What the agent can attest: all 8 previews built READY for 6a8541f (ci-status.sh, plus get_deployment on dpl_82ZTC3dizR1S3zxt7hfq7zYMxby3 confirming readyState: READY at sha 6a8541f).
  • production-readiness: run — 1 blocking-ish ⚠️ (gateway credential unconfirmed at runtime), plus design findings; see below.
  • code-review: medium (spec complexity: standard) — 4 findings, 3 fixed on branch, 1 raised.
  • security-review: run — no HIGH or MEDIUM findings. New surface is one parameterless, twice-gated GET returning a three-value enum; no user input, no injection surface, no SSRF (fixed gateway host), module-scope cache holds account-global data only, React with no unsafe render. Net effect is positive: the diff removes the DOM leak in the BRD chat.
  • playwright: TODO — manual DoD smoke performed instead

Preview caveat (recorded, not waved away)

The head that Ship inherits is the verify.md commit, which is markdown-only, so turbo-ignore will skip the app previews for it exactly as it did for 30e511d. The previews that matter were built for 6a8541f, which contains every line of code and docs in this run; the delta to the final head is this file alone. Smoke against the 6a8541f previews, not the final SHA's (which will not exist).

This is structural to the pipeline — the stage's own record is always the last commit — and worth fixing in the contract rather than in this run.

DoD smoke (on the preview — each line says who verified it)

Agent-run:

  • Route/gate wiring traced end to end in the diff — proxy.ts matcher covers /(api|trpc)(.*) deny-by-default, and route.ts:24-27 repeats the auth() check in-route (agent, by code reading — not demonstrated on a preview)
  • Response carries { status } and nothing else — asserted by whole-object equality in route.test.ts, so a leaked field fails the suite (agent; suite executed by CI's Run tests step inside Quality Project, green on 6a8541f)
  • unknown renders nothing — component state starts unknown and a failed or non-ok probe leaves it there (agent, by code reading)
  • Balance → verdict mapping pinned, including that an unreadable balance and a thrown probe both give unknown rather than degraded (agent; unit tests, green in CI)

Operator-demonstrated — all outstanding, none of these can be ticked yet:

  • AI_GATEWAY_API_KEY present on agentic-interface and web, Production and Preview — the headline item; see Findings F1 (operator)
  • Signed-in: conversation opens on the agentic preview with no warning while the account is healthy (operator)
  • Signed-in: forced degraded shows the warning above the composer and sending still works — devtools request interception, per the runbook (operator)
  • Signed-out caller is refused by the app — agent could not reach past Vercel SSO (operator)
  • BRD chat renders the fixed copy on failure, no provider text in the DOM (operator)
  • Runbook renders at /technical/ai-gateway on the docs preview (operator)
  • Open acceptance criterion: a live, non-canned BRD chat and onboarding chat each complete a turn — on preview and on production (operator)
  • auth: affected personas sign in and reach their dashboard (operator)
  • payments: not touched — no payment path in the diff (agent)
  • notifications: none expected — no template, no send, no notification surface in the diff (agent)

Findings & cleanup

Fixed on branch (6a8541f):

  • F2 — ai-availability.ts asserted "at or below zero the gateway declines every turn", and the runbook repeated it. Two passes independently flagged that this is unestablished on a pay-as-you-go plan with no cap: if the gateway keeps serving at a zero balance, degraded is a permanent false warning shown to every signed-in user. Fixed the claims, not the threshold — the mapping is unchanged and the decision is F5 below.
  • F3 — ai-availability.test.ts statically imported readAiAvailability without using it (all its tests go through freshProbe()). Dead binding against the 45-warning lint ceiling. Removed.
  • F4 — the runbook's "the in-product signal clears within 60 seconds" was wrong: that is the server-side cache. The conversation probes once on mount, so an open tab keeps the warning until reload. Corrected, and the once-on-mount behaviour is now stated rather than implied.
  • F6 — the probe's failure log now names the error class. A missing credential and a network blip both resolve to unknown and both render nothing, so the class was the only thing separating them — and the runbook's triage step depends on telling them apart.
  • F7 — the runbook claimed turbo.jsonglobalEnv covered the credential. That is a category error: globalEnv governs turbo's build env under strict mode and says nothing about what a serverless function sees at runtime. Rewritten to say the key must be set per Vercel project, Production and Preview, on both agentic-interface and web.

Outstanding — need an owner decision or an operator action:

  • F1 (operator action, blocks the last criterion). Nobody has confirmed AI_GATEWAY_API_KEY is actually set on agentic-interface. The spec and build notes reasoned from globalEnv, which is the wrong instrument (F7). The only record is a struck-through line in an archived run. And the probe cannot settle it: with no key the SDK falls back to OIDC, which throws when absent → unknown → nothing rendered. In precisely the misconfiguration this run exists to prevent, the signal is silent. One look at vercel env ls --scope sustentus settles it.
  • F5 (owner decision). What does this gateway do at a zero balance on uncapped pay-as-you-go? If it keeps serving, the degraded threshold is wrong and should move — the copy is already hedged, but showing every user a warning while AI works is worse than the outage. If it refuses, the current mapping is right as it stands.
  • F8 (owner decision, pre-existing contradiction this PR surfaces). The runbook records "Spend cap: None — deliberately" (Jamie's Q2 ruling), but apps/agent/app/api/chat/route.ts:76 justifies its deliberately fail-open rate limiter with "the AI Gateway project budget is the hard backstop on spend". There is no such backstop. Either a cap exists and the runbook is wrong, or the limiter's rationale is stale and should be corrected. Left untouched: it is a spend-control decision already ruled on once, and the file is outside this run's diff.
  • F9 (accepted, noted). The BRD chat fix moves the provider error from the DOM to the browser console. The text still crosses to the client — it arrives in the stream — so this is presentational containment, not redaction. True redaction needs server-side sanitising of the stream. Matches the existing concierge and agentic-chat pattern; out of scope here.
  • F10 (accepted, noted). /api/ai/availability has never executed in any deployed environment — zero runtime logs on every preview for this branch and on production. It is proven only by unit tests against a mocked gateway. The operator smoke above is its first real execution.
  • F11 (accepted, minor). verdictFromBalance's number and undefined branches are unreachable from the real SDK, whose schema requires balance: string. Harmless defensiveness, but four of its six tests exercise inputs the gateway cannot produce, so the file reads as more coverage than it is.

Context budget: within band. Beyond the Inputs table this stage read apps/agent/proxy.ts and packages/ui/src/lib/variants.ts (to check claims rather than assume them), and used the Vercel MCP to establish preview readiness after a plain fetch was refused by deployment protection.

05_ship/output/changelog.md


title: The assistant tells you when it cannot reply date: 2026-08-27T18:00:00Z personas: [customer, expert, csm, sdm, admin, vendor] slug: ai-gateway-billing-unlock pr: https://github.com/sustentus/sustentus/pull/900

The assistant tells you when it cannot reply

The Sustentus assistant now checks whether it can reach its model before you start typing. If it cannot, a short notice appears above the message box. Sending stays enabled — the check is a warning, not a lock — so you can still try, but you will not lose a carefully written question to a silent failure.

Nothing appears when the check itself cannot run. A test that has not answered is not evidence that the assistant is down, and we would rather say nothing than tell you it is broken when it is not.

Requirements conversations changed too. When the BRD chat cannot reach its model it now says so in plain words and invites you to try again, instead of showing the raw technical error it used to. Your lead stays exactly where it is, and you can pick the conversation back up when service returns.

05_ship/output/investor-update.md

Live AI is funded, and an outage now announces itself

Who it's for: every persona using the assistant, and customers writing a BRD What shipped: the AI Gateway runs on funded pay-as-you-go credits with a $100/month alert, the assistant warns before you type if it cannot reply, and the BRD chat no longer shows raw provider errors. Why it matters: unblocks the autonomous AI agents in Scale the Bridge.

Dig deeper: https://github.com/sustentus/sustentus/pull/900 · https://help.sustentus.com/changelog/2026-08-27-ai-gateway-billing-unlock

05_ship/output/release.md

Ship: ai-gateway-billing-unlock

  • pr: #900 · merge: authorised — Ready to merge ticked by Jamie; this commit rides the squash
  • CI: GREEN on 6a8541f at hand-off from Verify. This stage adds docs, the changelog page and the ship outputs, so that verdict is stale by construction — the merge rests on a fresh settled GREEN from ci-status.sh on the head this file rides, established after this stage's last push and covering the Vercel commit statuses as well as the check runs. Nothing merges on RED or PENDING.
  • technical docs: technical/ai-gateway (the runbook, added in Build) · technical/deployment (AI_GATEWAY_API_KEY added to the Agent variables, with the two-project / Production+Preview reality and why a missing key is silent) · technical/applications (the Agent section now names GET /api/ai/availability and what the conversation does with it) · technical/page.mdx (index link, added in Build)
  • business docs: business/service-journey/requirements — an AI unavailable row in the BRD agent's Exception Handling table, which is the user-visible half of this change: fixed copy, a retry, no provider or billing detail, and the lead left where it is.
  • release notes: both — ship note + changelog entry (apps/help/app/changelog/2026-08-27-ai-gateway-billing-unlock)
  • sent: queued — .github/workflows/ship-note.yaml fires on the merge that carries this file and emails 05_ship/output/investor-update.md to #product-update. Both Dig deeper links are filled; no placeholder remains.
  • close-out: archives this run to apps/docs/archive/pipeline-runs/ai-gateway-billing-unlock/. The ai-platform-enablement epic is not finished by this run — close-out.sh decides, and the epic's remaining stubs are its own record.

Acceptance check (vs spec)

  • GET /api/ai/availability answers a signed-in caller with ok / degraded / unknown and refuses a signed-out one — verified in Verify by code reading (proxy.ts edge matcher plus the in-route auth() check) and by route.test.ts, green in CI. Not demonstrated against a preview: Vercel SSO deployment protection refuses the agent.
  • The response never carries the balance, tier, gateway error text or a billing URL — pinned by whole-object equality in route.test.ts, so a leaked field fails the suite.
  • The probe result is cached server-side — ai-availability.test.ts pins both the reuse inside the window and the re-ask after it.
  • degraded shows a non-blocking notice before the user sends, and sending still works — verified in the diff; the composer is untouched by the notice.
  • unknown renders nothing — state starts unknown and a failed or non-ok probe leaves it there; the mapping is pinned by unit tests.
  • The BRD chat renders fixed copy, with the raw error going to the browser console — the error.message render is gone from apps/web/components/projects/brd-chat/brd-qualification-chat.tsx.
  • A billing runbook exists under technical/ naming the account owner, the pay-as-you-go arrangement, where the balance is read, the top-up path, the $100/month alert and the smoke-test procedure — apps/docs/app/technical/ai-gateway.
  • Open. A live, non-canned BRD chat and onboarding chat each complete a turn, on preview and on production. Not demonstrated. This is the criterion Verify recorded as the operator's, and it stays open through the merge — the ticked gate authorised the merge, it did not satisfy this line.

Carried past the merge (from 04_verify/output/verify.md)

These were raised at Verify, ruled on by the ticked gate rather than closed by it, and are recorded here so the merge does not bury them:

  • F1 (operator). AI_GATEWAY_API_KEY is unconfirmed on agentic-interface and web, Production and Preview. The probe structurally cannot settle it — no key falls back to OIDC, which throws, which reports unknown, which renders nothing. technical/deployment now says this in the place an operator actually looks; vercel env ls --scope sustentus settles it.
  • F5 (owner). What this gateway does at a zero balance on uncapped pay-as-you-go is unestablished. If it keeps serving, degraded is a false warning to every signed-in user and the threshold should move, not the copy. Shipped hedged (may not reply, sending stays enabled) rather than moved on a guess.
  • F8 (owner, pre-existing). The runbook records no spend cap; apps/agent/app/api/chat/route.ts justifies its fail-open limiter with "the AI Gateway project budget is the hard backstop on spend". One of the two is wrong. Outside this diff, deliberately untouched.
  • F10. /api/ai/availability has still never executed in a deployed environment. The operator smoke will be its first.

Context budget: within band. Beyond the Inputs table this stage read the four docs pages it changed and one prior changelog entry (for the frontmatter shape).