Run: source-analysis-pipeline

• branch: claude/source-analysis-pipeline-y380xf
• pr: #607

Stub: Source-analysis pipeline (website/description → draft blueprint)

• feature-slug: source-analysis-pipeline
• epic: tenant-onboarding-wizard
• personas: Admin, Vendor
• initiative: Build the Bridge / objective: Q2-2026 O1 — Establish PMF with Vendor Partners
• depends-on: ai-agent-foundation, onboarding-blueprint-model
• sequence: 4 of 7

Problem

The concierge's "wow" is that the tenant's catalogue drafts itself from something the vendor already has — a website URL, an uploaded price list, or a free-text description. Issue #588 Q2: there is no source-analysis capability today; the demo fakes it with scripted data.

Proposed change

An extraction pipeline in packages/services/src/ai (on the stub-2 foundation): fetch + parse the source (website scrape, uploaded file text, or raw description), then structured LLM extraction into a typed draft blueprint — the 7 sections with per-item confidence — written into the persisted blueprint from stub 1. A clear extraction contract (Zod schema shared with the blueprint model) is the deliverable alongside the pipeline. Graceful degradation when a source can't be analysed (unreachable site, thin content): return a partial or empty draft with the reason, so the concierge (stub 5) falls back to interviewing rather than erroring.

Acceptance criteria (rough)

• Given a reachable website URL, the pipeline produces a draft blueprint whose items carry confidence scores and populate the persisted blueprint for that tenant.
• A free-text business description (no URL) also yields a draft — same contract, lower expected coverage.
• An unanalysable source degrades to an explicit partial/empty result with a machine-readable reason — never an unhandled failure.
• Extraction output is schema-validated; malformed model output never reaches the blueprint document.

Out of scope (this feature)

• The conversational agent that decides when to run analysis (stub 5).
• Committing drafts to real services (stub 3).
• CRM connectors / OAuth imports (the demo's concept-B source list beyond URL + file + text).
• Any UI.

Notes for Define

Price-list upload can piggyback on Vercel Blob (already in the stack); Define should decide whether file parsing (PDF/CSV) is in this cut or URL+text only, flagging file support as a fast-follow if it inflates the PR. Website fetch must respect the outbound-proxy constraints of the deploy environment. touches: packages/services/src/ai/ (new onboarding/extraction or similar).

Spec: Source-analysis pipeline (source → draft blueprint)

• slug: source-analysis-pipeline
• personas: Admin, Vendor
• touches: packages/services/src/ai/onboarding
• complexity: complex

Problem

The onboarding concierge's "wow" is that a tenant's catalogue drafts itself from something the vendor already has — a website URL or a free-text description of their business. Today there is no source-analysis capability (issue #588); the apps/demo concierge fakes it with scripted data. The persisted blueprint (onboarding-blueprint-model) and the reusable agent core (ai-agent-foundation) now exist, but nothing populates a draft from a real source. This blocks the concierge agent (stub 5) and the wizard UI (stub 6), and it directly advances Build the Bridge / Q2-2026 O1 — establish product-market fit with vendor partners (KR: onboard 8+ vendors onto paid tiers) by making first-run setup feel effortless rather than like a data-entry chore.

Proposed change

Add an extraction pipeline under packages/services/src/ai/onboarding that turns a source into a typed draft blueprint and writes it into the persisted blueprint for a tenant. Functionally:

• Input. One of two source types this cut: a website URL (fetched + parsed to text) or a free-text business description (used directly). The public entry point is a single tenant-scoped service function, analyseSource({ tenantId, source }).
• Extraction contract. A single shared Zod schema — the canonical extraction contract — describing the draft across the seven existing blueprint sections (profile, categories, products, services, skills, sla, settings), each item carrying a 0–100 confidence score. The schema is the deliverable alongside the pipeline and is the type the LLM is generated against (via the foundation's generateStructured) so malformed model output can never reach the document.
• Persistence. Validated sections are written into the tenant's blueprint through the existing OnboardingBlueprintService.upsertSection (provenance ai_drafted, review state pending) — the pipeline creates no new model and no new collection.
• Graceful degradation. When a source can't be analysed — unreachable/blocked URL, thin or empty content, or a model failure — the pipeline returns an explicit partial-or-empty result with a machine-readable reason (a typed enum, e.g. unreachable / thin_content / extraction_failed) rather than throwing, so the concierge (stub 5) can fall back to interviewing.
• Environment. Website fetch goes through the deploy environment's outbound proxy and enforces a timeout and a response-size bound; it never blocks the caller indefinitely.

The pipeline is a service module only — no conversational logic, no commit-to-real-services, no UI.

Acceptance criteria

• Given a reachable website URL, analyseSource produces a draft whose items carry 0–100 confidence scores and are persisted into that tenant's blueprint sections (provenance ai_drafted, review state pending), readable back via OnboardingBlueprintService.
• A free-text business description (no URL) yields a draft through the same contract and the same persistence path (lower expected coverage is acceptable — it is not an error).
• An unanalysable source (unreachable/blocked URL, or thin/empty content) returns an explicit partial-or-empty result carrying a machine-readable reason — never an unhandled throw.
• Extraction output is validated against the shared Zod contract before any write; malformed or schema-violating model output is rejected and never reaches the blueprint document.
• Website fetch enforces a timeout and a response-size bound and routes via the outbound proxy; a slow or oversized source degrades to a reason, it does not hang.

Out of scope

• File/price-list upload (PDF/CSV) — deferred as a fast-follow. URL + text only this cut. The extraction contract is designed so a file source can be added later without changing it.
• The conversational agent that decides when to run analysis and interviews the user (stub 5).
• Committing drafts into the real services (stub 3, blueprint-commit-service — already built).
• CRM connectors / OAuth imports (issue #588 concept-B sources beyond URL + file + text).
• Any UI, route, or streaming surface (stubs 6–7).
• Re-analysing or re-onboarding existing tenants.

Open questions

• none. (Confidence bands, prompt copy, and the exact parser/library for HTML→text are Build implementation choices, not scope; the degradation-reason enum values above are indicative and may be refined in Build without changing the contract's shape.)

Build notes: source-analysis-pipeline

• commits: feat: source-analysis-pipeline — extract a draft blueprint from a URL or description

What changed

New server-only module packages/services/src/ai/onboarding/, on the existing AI foundation (generateStructured) and the persisted blueprint from stubs 1/3:

• contract.ts — the canonical extraction contract. DraftBlueprintSchema = the seven onboarding sections, each an array of confidence-scored DraftBlueprintItemSchema items (key / label / data / confidence 0–100). satisfies Record<OnboardingSectionKey, …> keeps the shape in lockstep with the model's section vocabulary; the item shape mirrors BlueprintItemInput so a validated draft item persists unchanged. Designed so a file source plugs in later unchanged.
• fetch-website.ts — fetchWebsiteText(url): standard fetch (so the runtime's outbound proxy governs egress), an AbortSignal.timeout (8 s) and a streamed 2 MB byte cap, protocol + content-type guards, dependency-free HTML→text. Any network/HTTP/size/content problem returns { ok: false, reason: "unreachable" } — it never throws or hangs.
• analyse-source.ts — the entry point analyseSource({ tenantId, source }). Resolves the source to text (URL or free text), runs structured extraction against the contract, and persists each non-empty section via onboardingBlueprintService.upsertSection (items land as ai_drafted / pending). Every failure degrades to an explicit result with a machine-readable AnalysisFailureReason (empty_source / unreachable / thin_content / extraction_failed) — status analysed (full), partial (thin source, some items) or empty.
• index.ts — module barrel.
• server/index.ts — re-exports analyseSource, the contract, and the types from @sustentus/services/server (server-only, alongside the catalogueTool precedent — network + DB access).

Acceptance criteria status

• Reachable URL → draft persisted with 0–100 confidence, ai_drafted/pending, readable back via OnboardingBlueprintService — analyseSource → upsertSection (defaults provenance/reviewState).
• Free-text description yields a draft through the same contract + persistence path — kind: "text" branch of resolveSourceText, identical downstream.
• Unanalysable source → explicit partial/empty result with a machine-readable reason, never an unhandled throw — every path returns; resolveSourceText and generateStructured failures map to reasons.
• Extraction validated against the shared Zod contract before any write; malformed output rejected — generateStructured (generateObject) validates against DraftBlueprintSchema and throws on mismatch → extraction_failed, nothing persisted.
• Website fetch enforces a timeout + response-size bound and routes via the outbound proxy; degrades rather than hanging — AbortSignal.timeout + readCapped (2 MB) over standard fetch.

Verify result

• mechanical checks (format · lint · typecheck · build) run in CI + the Vercel preview, not here. Release reads them back from the PR's check runs. No check is expected to fail: lib includes DOM + DOM.Iterable and types: ["node"], so fetch / ReadableStream / AbortSignal.timeout / Buffer all resolve; undici was intentionally not added (not resolvable in the workspace).

Notes for review

• Proxy handling is deliberately implicit: standard fetch uses the runtime's default egress, so the deploy proxy governs it — writing a custom socket/dispatcher would bypass that. The real safety guarantees (timeout + size cap) are explicit in code.
• data payload is z.record(z.string(), z.unknown()) — intentionally loose, since each section's payload shape varies (pricing, tiers, locations). The model is steered by the system prompt's per-section guidance rather than a rigid per-section schema, to keep the contract stable.
• Section item cap is enforced by truncating to MAX_ITEMS_PER_SECTION at persist time (not via a schema .max), so an over-eager model degrades gracefully instead of failing validation.

A vendor's catalogue now drafts itself from their own website

Who it's for: Vendors and the admins onboarding them. What shipped: The onboarding concierge turns a vendor's website URL or business description into a confidence-scored draft catalogue. Why it matters: Removes the blank-page setup barrier — progress toward our Q2 objective, establish product-market fit with vendor partners.

Unusable or thin sources degrade to a clear reason, so onboarding never dead-ends.

Dig deeper: <merged-PR URL>

Release: source-analysis-pipeline

• pr: https://github.com/sustentus/sustentus/pull/607 · merged: pending — awaiting Ready-to-merge tick
• CI: green — Quality Project (format/lint/typecheck/build), preview DB migrate, and "Review diff against CONVENTIONS.md" all passed
• technical docs: updated apps/docs/app/technical/packages/services/page.mdx (AI section — new ai/onboarding/ source-analysis capability + /server export) in this PR
• business docs: no business docs impact — internal capability, no user-facing surface changed (the concierge/wizard that would expose it is epic stubs 5–6, out of scope this run)
• release notes: investor-only — no end-user changelog entry (internal change, no user-facing behaviour yet), matching the sibling foundation stubs in this epic
• deploy: pending
• sent: pending

Review summary

• The Claude Code Review action ("Review diff against CONVENTIONS.md") ran on the diff and passed with no inline comments (get_review_comments → 0 threads). Diff is small, additive, self-contained.
• Self-review for cleanup: no dead code, stray debug, or leftover scaffolding; all functions are arrow, type not interface, braces on every block — nothing to fix on the branch.
• Degradation is total by construction: every source/extraction failure path returns an explicit AnalysisFailureReason; the only throw (generateObject on schema mismatch) is caught and mapped, so nothing malformed reaches the blueprint document.

Acceptance check (vs spec)

• Reachable URL → confidence-scored draft persisted (ai_drafted/pending), readable via OnboardingBlueprintService — analyseSource → upsertSection.
• Free-text description yields a draft through the same contract + persistence path — kind: "text" branch, identical downstream.
• Unanalysable source → explicit partial/empty result with a machine-readable reason, never an unhandled throw — all paths return a typed result.
• Extraction validated against the shared Zod contract before any write; malformed output rejected — generateStructured/generateObject validates against DraftBlueprintSchema; throw → extraction_failed.
• Website fetch enforces a timeout + response-size bound and routes via the outbound proxy; degrades rather than hanging — AbortSignal.timeout(8s) + readCapped(2MB) over standard fetch.