Architecture of a multi-step AI assistant

A turn looks like this. A message arrives over a WebSocket connection. A classifier decides what the user is asking for. One of thirteen skills takes the turn from there, gathers what it needs, composes an answer, and that answer streams back token by token while the model is still producing it.

Every interesting decision is in the seams between those steps.

Why a classifier and not one big prompt

The alternative is a single prompt describing every capability with every tool bound to it, and the model deciding what to call. For a handful of capabilities that is the right answer and it is less machinery. Three arguments for routing show up as the count grows.

Prompt cost scales with capability count, on every turn. With one big prompt, a user asking to cancel a meeting still pays for the description of every skill they are not using. With a classifier, routing carries a short prompt with a constrained output, and then exactly one skill's prompt is loaded.

Behaviour stays isolated. In one large prompt, instructions interact: tightening the wording that makes one capability behave changes another capability that was working, and you find out in production. Thirteen prompts that each know only their own job do not have that property. That matters more than the token cost. It is the difference between a system you can change and one you can only add to.

Failures become localisable. When an answer is wrong in a single-prompt system, "why" has too many answers at once: retrieval missed, or the model picked the wrong tool, or instruction eleven contradicted instruction four. Splitting the first decision out lets you ask a cheaper question first: did it route correctly? That is a classification problem, with labels and a confusion matrix, testable without judging free text.

When the classifier is wrong

The two failure modes are not symmetric, and only one is dangerous.

Routing to the wrong skill is loud. The user asked to move a meeting and got an answer about booking a new one. It is obviously wrong to the person reading it, so they rephrase.

Falling through to the general conversational skill is quiet. That skill can answer almost anything plausibly, so a misroute here does not look like a misroute. It looks like a mediocre answer, and a system that is confidently vague erodes trust faster than one that is visibly wrong.

So the fallback has to be honest rather than helpful. The general skill is the retrieval-backed one, and the composer is instructed to ground every fact in what retrieval actually returned, and to say it does not have the information rather than fill the gap. An assistant that says "I do not have that" is recoverable; one that improvises is not.

The other thing that follows is that the conversation is the repair channel. The turn carries a window of recent messages, so a follow-up like "no, I meant the one on Thursday" reclassifies with the previous turn in view.

That is worth making structural rather than leaving to luck. The router does not only return a label; it returns a confidence score with it, and below 0.55 the turn produces a clarifying question instead of an answer. The fallback path reports a confidence of zero, so anything the router could not place lands in the same branch.

This is a different mechanism from the version of a confidence threshold that does not work, and the distinction is the point. A threshold that suppresses answers is a bad idea: a model is confidently wrong often enough that suppression mostly costs you correct answers while the confident errors sail through untouched. A threshold that converts low confidence into a question is not judging whether the answer is good. It decides whether the system has enough to go on, and when it does not, hands the turn back to the person who does.

Speculative, parallel context fetching

The obvious sequencing is: classify, hand off to a skill, let the skill ask for what it needs, fetch it, generate. That is correct and slow: it is strictly serial, every fetch is a round trip, and the user watches a spinner for the sum.

The change is to stop waiting for permission. At the start of the turn, before the classifier has said anything, four fetches go out in parallel: recent conversation history, the user's profile, the tenant's meeting types, and calendar events in a window of the next thirty-five days. Retrieval against the knowledge bases is a fifth, issued separately and itself doubled, because there is a personal store and a workspace store. When a skill takes over it consumes the futures it needs, and the rest are never read.

It is not a flat fan-out. Selecting calendar events needs a timezone, and the timezone comes from the profile, so that branch is chained onto the profile fetch rather than started cold. What you build is a small dependency graph: four branches, one of them two-stage, each starting as early as its inputs allow.

The effect on latency is the point. Turn time goes from

classify + fetch_a + fetch_b + fetch_c + generate

to

max(classify, fetch_a, fetch_b, fetch_c) + generate

Because classification runs concurrently with the fetches rather than in front of them, routing costs almost nothing in wall-clock terms. You get the isolation of a separate classifier without paying a round trip for it, and that is what makes the split worth doing.

When speculation does not pay

It is waste by construction: most of what is fetched is thrown away. It is a good trade here for three reasons. The fetches are reads and idempotent, so nothing observable happens if a result is discarded. The fan-out is bounded and fixed in the code rather than chosen by the model, so a turn cannot decide to issue forty requests. And the generation call dominates the cost of the turn by a wide margin.

Change any of those and it stops being a good idea:

• Anything with a side effect. Never speculate on a write. Obvious, and exactly what arrives by accident when a read grows a cache-warming write inside it.
• Rate-limited third-party APIs. Speculating a CRM lookup every turn spends quota on turns that never needed it, and the failure shows up as throttling during a busy hour rather than as a slow turn. Those stay lazy.
• Expensive fetches with a low hit rate. A fetch used by one skill out of thirteen, which is also the slowest, shortens nobody's critical path.

The rule that survived: speculate when the fetch is read-only, bounded, and wanted by a decent share of the skills. Otherwise leave it lazy.

What streaming breaks

You cannot validate what you have already sent. Any check over a complete answer has to move in front of the first token or become a visible correction after the fact. Deciding which checks are cheap enough to run before generation is something streaming forces on you and a buffered response does not.

Failure mid-stream has no natural shape. With a normal response you have a status code, decided before the first byte. Once tokens are flowing, a failure is indistinguishable from an answer that stopped, so the protocol needs two explicit terminal events, finished and failed. Without them the client's only signal is silence, and silence is ambiguous. This is the same class of problem as putting a streaming boundary above a route that can return a 404: the status goes out with the first flush, so anything that wants to change it has to happen before that.

The token is not the right unit to send. A frame per token costs a message and a redraw per token, and that is where ragged rendering comes from. The output goes through a coalescing buffer instead: hold until sixty characters, flush on the first sentence or clause boundary after that, force a flush at two hundred and forty so a long unbroken run cannot stall.

Renderers see half-parsed markup. A code fence opens dozens of tokens before it closes; a link is a broken bracket for a moment. The renderer has to tolerate an incomplete tree rather than assume valid input, and flushing on boundaries keeps the invalid intermediate states rare instead of one per token.

Resuming mid-stream is the thing we did not build. There is no sequence number and no replay. If the connection drops during generation the client reconnects and reloads the stored conversation: completed turns come back from persistence, a turn that was still generating is gone. The cost lands where the user is least patient, on a long answer over a bad connection. Doing it properly means numbering the stream, persisting the partial turn as it is produced, and letting a returning client continue from an offset. That is a protocol change rather than a buffer change, which is why it did not happen alongside the rest.

What I would do differently

The router has a confidence threshold but no labelled dataset and no tracked confusion matrix behind it, and that pairing is the gap. A threshold is a number somebody chose. Without a measured matrix you cannot say what it costs you, which makes it a guess that happens to work rather than a tuned parameter. It deserved a labelled set from the first week: cheap early, expensive to retrofit.