Cookbook: Parallel questions

TL;DR Every question is scored on its own against the document, so batching N questions into one system_one call changes neither the mean nor the run-to-run std dev of any answer. It changes the bill: on a ~54,000-character article, 13 questions in one call cost $0.000497 in 0.27s versus $0.006090 in 2.71s for 13 separate calls — 12.2x cheaper, 10.0x faster.

Goal

Settle the question "does putting N questions in one request change the answers?" empirically, and quantify what batching saves. The experiment asks each question 5 times both ways — all N in one request, and one question per request — and compares the run-to-run standard deviation: how far an answer moves from one repeat to the next.

The case is a regulatory briefing: a compliance team wants 13 things checked against the GDPR article.

Inputs / state shape

The state is a dict wrapping a pinned Wikipedia revision:

WIKIPEDIA_REVISION = 1363040264  # "General Data Protection Regulation", as of 2026-07

DOCUMENT = {
    "source": f"https://en.wikipedia.org/?oldid={WIKIPEDIA_REVISION}",
    "text": fetch_article(WIKIPEDIA_REVISION),
}

and each call sends state={"article": DOCUMENT}. The article is 53,777 characters — a document-dominated workload where the document is most of every request. The revision is pinned and cached so the numbers stay fixed as the live article is edited.

Constants: TYPESAFE_MODEL = "jev-1.12", PRICE = (0.042, 0.00) ($ per 1M input/output tokens, jev-1.12 as of 2026-09), RUNS = 5.

Questions asked

Thirteen questions: 8 Noul, 2 Choice, 3 Score. Verbatim.

Nouls (instructions only, no criteria):

Choices (criteria maps each label to its meaning):

"instrument_type": Choice(
    instructions="What kind of EU legal instrument is the GDPR?",
    criteria={
        "Regulation": "Directly binding law in all member states, no national implementation needed.",
        "Directive": "Sets goals that member states implement through national law.",
        "Treaty": "An international treaty between states.",
        "Recommendation": "Non-binding guidance.",
    },
),
"max_fine": Choice(
    instructions="What is the maximum administrative fine for the most serious infringements?",
    criteria={
        "TwentyM_or_4pct": "Up to EUR 20 million or 4% of annual worldwide turnover, whichever is greater.",
        "TenM_or_2pct": "Up to EUR 10 million or 2% of annual worldwide turnover, whichever is greater.",
        "FixedCap": "A fixed amount not tied to turnover.",
        "NoFines": "The GDPR provides no administrative fines.",
    },
),

Scores (criteria lists level descriptions, from level 0 up — already the 0.6.0 ordered-sequence form):

"individual_rights": Score(
    instructions="How strong are the rights the GDPR grants to individuals over their data?",
    criteria=[
        "None: individuals get no rights over their data.",
        "Weak: a right to be informed, but little control.",
        "Moderate: access and correction rights, but limited means to act on them.",
        "Strong: access, erasure, portability, and objection rights, with enforcement behind them.",
    ],
),
"penalty_severity": Score(
    instructions="How severe are the penalties the GDPR provides for non-compliance?",
    criteria=[
        "None: no penalties of any kind.",
        "Symbolic: small fixed fines unlikely to change behavior.",
        "Substantial: fines large enough to matter to most companies.",
        "Severe: fines scaled to global revenue, material even to the largest companies.",
    ],
),
"compliance_burden": Score(
    instructions="How heavy is the compliance burden the GDPR places on organisations?",
    criteria=[
        "Negligible: no meaningful obligations.",
        "Light: a few notices and disclosures.",
        "Moderate: documented processes and some dedicated roles for larger processors.",
        "Heavy: records, impact assessments, officers, and breach procedures for many organisations.",
        "Extreme: obligations so demanding that ordinary organisations cannot fully comply.",
    ],
),

One number is tracked per answer, by type: Noul → the probability of "yes"; Choice → the max prob, the probability on the picked label; Score → the score normalized to 0–1, the score divided by the top level.

Combining logic in code

import os
from statistics import mean, stdev
from time import perf_counter

from typesafe_sdk import Choice, ChoiceAnswer, Noul, NoulAnswer, Score, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
PRICE = (0.042, 0.00)
RUNS = 5
client = TypeSafeClient(api_key=os.environ["TYPESAFE_API_KEY"], timeout=120.0)


def ask(keys: tuple[str, ...], run: int):
    """One TypeSafe call -> ({key: tracked metric}, input_tokens, output_tokens, latency_s);
    ``run`` only forces a distinct live call per repeat."""
    started = perf_counter()
    response = client.system_one(
        state={"article": DOCUMENT},
        questions={key: QUESTIONS[key] for key in keys},
        model=TYPESAFE_MODEL,
    )
    values = {}
    for key in keys:
        answer = response.answers[key]
        if isinstance(answer, NoulAnswer):
            values[key] = answer.noul
        elif isinstance(answer, ChoiceAnswer):
            values[key] = max(answer.probabilities.values())
        else:
            values[key] = answer.score / (len(QUESTIONS[key].criteria) - 1)
    return (
        values,
        response.usage.input_tokens,
        response.usage.output_tokens,
        perf_counter() - started,
    )


def priced(result):
    """({key: metric}, in_tokens, out_tokens, latency) -> ({key: metric}, cost_usd, latency)."""
    values, input_tokens, output_tokens, latency = result
    return values, input_tokens / 1e6 * PRICE[0] + output_tokens / 1e6 * PRICE[1], latency


batched = [priced(ask(tuple(QUESTIONS), run)) for run in range(RUNS)]
singles = [{key: priced(ask((key,), run)) for key in QUESTIONS} for run in range(RUNS)]

Note answer.score / (len(QUESTIONS[key].criteria) - 1): the normalizer is the top level index, so a 4-level Score divides by 3 and the 5-level compliance_burden divides by 4. This is one of the places where the 0.6.0 ordered-sequence criteria makes the arithmetic obvious.

Cost is applied after the cached call, so a price change needs no new API calls.

Results / what the cookbook reports

Answers (5 runs per strategy). Mean and std dev of each question's tracked number:

Choices, scores, and six of the eight nouls came back identical across all 5 repeats under both strategies. breach_72h and criminal_penalties carry a little run-to-run sampling noise, the same size either way, with means agreeing to within that noise — the noise is a property of the question, not of how you batch.

Cost and speed:

batching                 calls        cost  total time
one call, all 13             1   $0.000497       0.27s
13 calls, one each          13   $0.006090       2.71s

batching: 12.2x cheaper, 10.0x faster

The speed figure sums the 13 single-call latencies, so it assumes they run one after another. Fire them concurrently and the gap shrinks, but the 13x token cost stays. The bigger the document, the nearer the saving comes to a full Nx.

Adapting it to a new domain

1. Put the shared document in state once and give every question a distinct key.
2. Keep the per-type metric extraction (answer.noul, max(answer.probabilities.values()), answer.score / (levels - 1)) so heterogeneous questions land on one comparable scale.
3. To audit stability in your own domain, re-run the same batched call RUNS times with a run argument that only exists to defeat caching, and report per-question std dev.
4. Price after the fact from response.usage.input_tokens / output_tokens rather than baking a rate into the call path.

Gotchas

• The saving is document-dominated. With a short state and many long questions, batching saves much less; the article here is 53,777 characters against 13 short instructions.
• Latency comparison is sequential by construction. Concurrent single calls narrow the wall-clock gap but not the token bill.
• Not every answer is noise-free. Two of thirteen questions moved run to run; if you need a stable number near a decision threshold, measure your own question rather than assuming std dev 0.
• Choice normalization. The tracked Choice metric is the max probability, which will read 1.000 on unambiguous questions and tells you nothing about which label won — keep answer.choice too if you care.
• Install line. pip install ipython "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/. cooksafe is a helper package on TypeSafe's private index, and pypi.typesafe.ai returned 404 publicly as of 2026-09-17 — install pip install typesafe-sdk and reimplement the two helpers: JsonCache(Path("json_cache.json")), a decorator memoizing each call's return value into a JSON file keyed on its arguments (delete to re-run live), and make_playground_link(state, questions, models=[...]), which packs the state and questions into a https://console.typesafe.ai/playground#share/... URL.
• Published run used jev-1.12; this wiki documents jev-1.13.0.

Related

• Speculative fan-out — the pattern this cookbook measures
• Primitives: Choice, Score, Noul — Noul, Choice and Score side by side
• Score questions — level criteria and score normalization
• Models, aliases, pricing, rate limits, context — the token rate behind PRICE
• Python SDK responses, answers, usage, models — response.usage, NoulAnswer, ChoiceAnswer
• Cookbook: Re-ranking — a cookbook that deliberately asks one question per pair, and says so
• Cookbooks overview — the full cookbook catalog

Sources

• raw/docs/cookbooks__parallel_questions.md (https://docs.typesafe.ai/cookbooks/parallel_questions.md)