---
title: "Speculative fan-out"
type: pattern
tags: [patterns, fan-out, speculative, latency, cost]
created: 2026-09-17
updated: 2026-09-21
confidence: high
sources:
  - raw/docs/patterns__fan-out.md
  - raw/docs/patterns.md
  - raw/github/skills/skills/typesafe-ai/SKILL.md
jev_version: "jev-1.13.0"
summary: "Put every question your decision tree could need into one call, including speculative ones, and let code discard the irrelevant answers."
---

# Speculative fan-out

> **TL;DR** Send all the questions your system could need in a single `POST /v1/systemone` call — including ones that only matter on branches you have not taken yet — then branch in code. "All questions are evaluated in parallel, so adding more questions usually has little effect on response time."

## Problem

A decision tree has dependent branches: you classify a support ticket, and *only if* it is a bug report do you need its severity and whether it has reproduction steps; *only if* it is billing do you need to know whether a refund was requested.

The obvious implementation asks the classifying question first, waits, then issues a second call for the branch-specific questions. That serializes two round trips for what is logically one decision, and the second call re-sends the same state.

## Pattern

From `raw/docs/patterns__fan-out.md`:

> Because TypeSafe supports sending many questions in a single API call, we recommend putting all of the questions your system needs in a single request, and then using code to decide what is relevant after the fact. All questions are evaluated in parallel, so adding more questions usually has little effect on response time.

> **Wording changed 2026-09-21.** Until this refresh raw/docs/patterns__fan-out.md said adding questions "typically doesn't add **any** latency to the response", and the speculative-questions note said extra questions cost "no speed cost". Both were softened to "usually has little effect on response time". The pattern is unchanged; the guarantee is not as absolute as it read before, so measure end-to-end latency on your own question set rather than assuming extra questions are free.

A **speculative question** is one you ask before you know whether its answer will be used. The source's definition:

> **Speculative questions:** `bug_severity` and `has_reproducible_steps` only matter if the ticket is a bug report. `refund_requested` only matters for billing. We include all upfront because additional questions usually have little effect on response time. If the ticket turns out to be a feature request, the bug severity result will be irrelevant, in which case your code path simply ignores it.

Two steps: fan out, then route with code.

## Implementation

### Step 1: speculative fan-out

The documented example is support ticket triage. State and questions, verbatim from the source (the source renders these as a playground example; `state` and `questions` are the two top-level fields of the request body — add `"model": "jev-latest"` to make it a complete HTTP request, see [[reference/http-api]]):

```json title="request"
{
  "state": "Hi, I placed an order (#98423) last Thursday and was charged twice. I also can't log in after the site update, and adding Apple Pay would be really helpful. This is getting frustrating.",
  "questions": {
    "category": {
      "type": "choice",
      "instructions": "Determine the broad category of this support ticket",
      "criteria": {
        "bug_report": "The user is reporting something that is broken or producing errors",
        "billing": "Charges, invoices, refunds, subscriptions",
        "feature_request": "The user is requesting new functionality",
        "account": "Login, permissions, profile, security"
      }
    },
    "bug_severity": {
      "type": "score",
      "instructions": "How severe is the reported issue",
      "criteria": [
        "Cosmetic; no impact to functionality",
        "Broken or degraded feature; workaround exists",
        "Blocking issue; no workaround exists"
      ]
    },
    "has_reproducible_steps": {
      "type": "noul",
      "instructions": "The user describes specific steps to reproduce the issue"
    },
    "refund_requested": {
      "type": "noul",
      "instructions": "The user is explicitly asking for a refund or credit"
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated the user appears",
      "criteria": ["Calm, matter-of-fact", "Frustrated but civil", "Very angry"]
    }
  }
}
```

Note the mix: one Choice, two Scores, two Nouls — all in one call. See [[concepts/choice]], [[concepts/score]], [[concepts/noul]].

### Step 2: route with code

> Your code decides what is relevant based on the classification result:

```python title="triage.py"
category = response.answers["category"]
bug_severity = response.answers["bug_severity"]
bug_repro = response.answers["has_reproducible_steps"]
refund = response.answers["refund_requested"]
frustration = response.answers["frustration"]

if category.choice == "bug_report":
    if bug_severity.score > 1.5 and bug_repro.noul > 0.6:
        escalate_to_engineering(ticket_id, severity="high")
    else:
        add_to_bug_backlog(ticket_id)

elif category.choice == "billing":
    if refund.noul > 0.7:
        route_to_billing_with_flag(ticket_id, refund_likely=True)
    else:
        route_to_billing(ticket_id)

elif category.choice == "feature_request":
    log_feature_request(ticket_id)

# Frustration is useful regardless of category
if frustration.score > 1.5:
    flag_for_priority_response(ticket_id)
```

The source's closing point: "Everything needed for the full decision tree comes from one call. Speculative questions are ignored when irrelevant and save a round trip when they are not."

Note that `frustration` is not speculative — it is consumed on every branch. Fan-out mixes both kinds freely.

### Sending it

The example above is JSON for `POST /v1/systemone`. To run the same fan-out through an SDK, build the same `questions` map with `Choice` / `Score` / `Noul` (Python) or `choice()` / `score()` / `noul()` (JS) — see [[guides/quickstart]] for a complete runnable call, and [[reference/python-sdk]] / [[reference/javascript-sdk]] for signatures.

## When it fails

- **The next question depends on the previous *answer*, not just a branch.** The agent skill draws the line: "A second request is warranted when an earlier answer is needed to fetch evidence, construct new state, or determine the next options." Fan-out only collapses trees whose questions can all be written against the state you already have.
- **A speculative question with an unstated premise.** `bug_severity` is asked of tickets that may not describe a bug. The skill's rule: "State each speculative premise explicitly; code consumes the applicable answers." A question that silently assumes its branch produces meaningless numbers on the other branches — which is fine while you ignore them, and a bug the moment you do not.
- **Tokens are not free, and latency is only nearly free.** "Adding more questions usually has little effect on response time" is a latency claim, not a cost claim. The skill: "Extra questions still use tokens; measure actual request budgets, cost, and end-to-end latency." The `usage` object reports `input_tokens` and `output_tokens` per call; see [[reference/models-and-pricing]].
- **Reading a speculative answer's confidence as a system-health signal.** Low confidence on an unused branch means nothing. The skill: "Ignore uncertainty on unused branches."
- **Very wide fan-outs.** The pattern page gives no ceiling on question count; check [[reference/models-and-pricing]] for the context window and [[reference/rate-limits-and-errors]] before fanning out to dozens of questions (inferred — the source states no limit).

## Variants

- **Fan-out plus a confidence gate.** [[patterns/intent-routing]] asks `intent` and `complexity` together and gates on `intent.confidence` before consuming either.
- **Fan-out plus an LLM fallback.** [[guides/smart-home-demo]] fans out a long question list per utterance and hands the request to an LLM only when the classification says the user wants conversation.
- **Fan-out as typed argument filling.** The agent skill's "Route and fill known arguments" direction: "A request can select a handler and its typed parameters. Ask useful branch-specific questions up front and consume only the relevant answers." See [[cookbooks/function-calling]].
- **Fan-out over independent items.** The same call shape applied per document, in parallel; see [[cookbooks/parallel-questions]].

## Related

- [[patterns/overview]] — the four-pattern catalog
- [[patterns/intent-routing]] — classification plus routing, built on this pattern
- [[concepts/state]] — the payload every fanned-out question shares
- [[concepts/how-to-build]] — decomposing a workflow into atomic questions
- [[reference/http-api]] — the `questions` map on the wire
- [[guides/smart-home-demo]] — fan-out in a running demo

## Sources

- raw/docs/patterns__fan-out.md (https://docs.typesafe.ai/patterns/fan-out)
- raw/docs/patterns.md (https://docs.typesafe.ai/patterns)
- raw/github/skills/skills/typesafe-ai/SKILL.md (https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md)