$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/guides

Testing and evaluating a Jev workflow

[ guide ][ updated 2026-09-21 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#testing · evaluation · calibration · consistency · thresholds

TL;DR Build a small labelled set, then measure five things: accuracy against your labels, calibration (accuracy per confidence bucket), repeatability across repeats of the same input, latency, and cost from usage.input_tokens. Pick thresholds from the calibration table, not from a cookbook. Regression-test the jaggedness edge cases. And note before you publish: MCA §2.3(f) prohibits publishing benchmarks about the Services.

What to measure

Metric How Why it matters
Accuracy Agreement between your workflow's final action and a label per case The only metric that speaks to your application. Note TypeSafe's own "accuracy" means agreement with an ensemble reference, not truth (Workflow evals: how TypeSafe measures Jev).
Calibration Bucket answers by confidence (or by noul) and report accuracy per bucket Calibration is a property of groups, never of one answer (Confidence vs probability). This table is what sets your thresholds.
Consistency / repeatability Run the same input N times; report per-question standard deviation and how often the decision flips Jev is not deterministic (Cookbook: Self-consistency — nouls).
Latency Wall clock around the call, measured from where your service runs TypeSafe's published figures are measured "from our laptops on the West Coast" (Workflow evals: how TypeSafe measures Jev).
Cost response.usage.input_tokens / 1e6 * 0.042 Input tokens only; output tokens are free (Models, aliases, pricing, rate limits, context). Price after the fact, so a rate change needs no new calls.

Measure the workflow's decision, not just each answer. A question can wobble while the composed action stays put — and a rock-steady question can feed a decision that flips because the threshold sits exactly where the spread lands.

Building a labelled set

  1. Collect real inputs, including the boring majority and the genuinely hard cases. The consistency cookbooks each use a single input "built to sit on the fence"; a production set needs both kinds.
  2. Label the outcome you act on — the queue, the route, the approve/hold/deny — not the intermediate probabilities.
  3. Split tuning from validation. Tune wording and thresholds on one slice, validate on a held-out slice (Writing instructions and criteria that Jev reads correctly). Higher confidence on the tuning slice does not establish that an answer is right.
  4. If you have no labels, TypeSafe's own move is an ensemble of expensive reasoning models answering every question in the harness — its workflow evals use the average of GPT-6 Astra and Claude Fable 5.1 at high thinking as the reference (Workflow evals: how TypeSafe measures Jev). Carry the caveat with the numbers: a reference built from two competitors biases toward them, and the model being scored can never beat the reference by construction.
  5. Record the label's provenance. The classification cookbook is explicit that its gold labels are self-reported SIC codes filtered so the filing text supports them, so the numbers "measure the recipe rather than the state of EDGAR's metadata" (Cookbook: Classification using confidence).

Fifty to a few hundred cases is enough to start: the classification cookbook's headline result rests on 60 filings.

Using confidence buckets to find the threshold

Score every case, sort by confidence, bucket, and report accuracy per bucket. The worked example: one Choice over 75 SIC groups, confidence >= 0.9 on 30 of 60 filings was 27/30 (90%) correct, while the 30 below were 12/30 (40%) at group level and 70% when reported one level up the taxonomy (Cookbook: Classification using confidence).

Read three things off that table:

Two mechanical warnings. Gate a Noul on noul itself — Noul answers have no confidence field. And where a specific statistical rule applies, use probabilities rather than confidence; the choice-consistency cookbook states outright that its rule "uses the returned probabilities, not the API's separate confidence field."

A 50-case check, then shadow mode (community method)

A short go/no-go procedure from Nate B. Jones's member guide (community tier, used with his permission). It complements the bucket method above: use it first, when you have little labelled data.

  1. Pull about 50 real past cases where you already know the right answer.
  2. Run them through the Jev questions. Save the picked answer and its confidence for every case.
  3. Look at every miss and write down its confidence. Compare with what your current process did on the same 50.
  4. Set the threshold in the gap between the confidence of your hits and the confidence of your misses. In his 12-message support test the one miss came back at 0.53 and every hit at 0.75 or higher, so any gate between the two worked (his numbers, tiny set).
  5. No clean gap: this decision is not a fit yet. Split the question into smaller ones, or leave it with the LLM.
  6. Misses at high confidence point at the question, not the threshold. Check for two options that overlap (his example: 16GB and 16 GB offered separately split the probability) or one question that is really two (Writing instructions and criteria that Jev reads correctly).
  7. Wire it in shadow mode: Jev runs on live traffic and logs input id, answers, confidences and what the existing process did, while the old path still makes the real decision. Compare after about a week; switch over only if the log holds up. Keep the old path as the low-confidence fallback (Confidence-gated routing).

Self-consistency tests

The two consistency cookbooks are the method, not just results. Both hold one input fixed, run the rubric 15 times, and report per-question probability standard deviation, how often the top answer changes, and cost and latency per call.

Cookbook: Self-consistency — nouls Cookbook: Self-consistency — choices
Rubric 14 Nouls (claims triage) 8 Choices (moderation)
Repeats 15, plus six LLM conditions 15, plus six LLM conditions
Mean probability std dev 0.0102 0.0098
Per call 111 ms, $0.000043 114 ms, $0.000046
What moved covered spanned 0.43–0.53, crossing a 0.5 threshold 2 of 8 questions changed their top label
Mitigation Band: < 0.30 no, 0.30–0.70 uncertain → human, > 0.70 yes Abstain to uncertain below a 0.60 top probability

Four harness mechanics worth copying verbatim:

The actionable output is a list of questions whose spread straddles a decision threshold. Widen the band around them, re-word them, or decompose them — a question that abstains on every repeat is telling you the rubric is under-specified, not that the model failed.

Comparing against an LLM

system-one-adapter answers the same typesafe_sdk questions with an OpenAI or Anthropic model and returns a SystemOneResponse subclass, so your state, questions, and composition code stay byte-identical and only the client swaps (system-one-adapter: LLM-backed drop-in for TypeSafeClient):

from system_one_adapter import SystemOneAdapterClient

client = SystemOneAdapterClient(
    structured_outputs=True,
    llm_answer_mode="probabilities",
    normalize_probabilities=True,
)
response = client.system_one(STATE, QUESTIONS, provider="anthropic", model="claude-haiku-4-5")
print(response.usage.latency, response.usage.input_tokens_total, response.usage.n_retries)

It adds usage.latency, input_tokens_total / output_tokens_total across retries, retry counts, and per-attempt traces in response.debug["llm_attempts"] — the raw material for a like-for-like cost, speed, and agreement comparison. Price the LLM side with that provider's rates; the cookbooks' TYPESAFE_PRICE = (0.042, 0.00) covers the Jev side only.

Regression tests for jaggedness edge cases

Keep a standing suite of cases aimed at the documented failure modes (Jev 1.13 jaggedness: known failure modes), run on every question edit and every model bump:

Edge case Test
Literal reading An input that satisfies the letter of the instruction but not the intent — and vice versa.
Negation The question and its negation on the same input. They are not complements: one documented pair summed to 1.19. Assert your code never relies on that.
Numbers An input whose correct answer requires counting or arithmetic. The right result is that your code produced it, not a question.
Dates Mixed formats, relative references ("last Thursday"), quarter and window boundaries. Assert the parts came from Choices and the comparison from datetime.
Adversarial content State containing an injected instruction, a misleading framing, or text arguing for its own classification. Jev does not treat state as hostile by default.
Indirection A question about a property of a property; confirm the flattened rewrite scores better.
Cross-type invariants If any code compares a Noul to a Choice probability, or a threshold tuned on one type to the other, delete it.

Also pin the model. Thresholds tuned against jev-latest shift silently when the alias moves; pin jev-1.13.0 and re-run the suite before adopting a new version (Models, aliases, pricing, rate limits, context).

A minimal harness

Runnable with pip install typesafe-sdk (0.6.0) and TYPESAFE_API_KEY set. Replace CASES and QUESTIONS with your own.

"""Minimal Jev evaluation harness: accuracy, calibration, repeatability, latency, cost."""

import statistics
from collections import Counter, defaultdict
from time import perf_counter

from typesafe_sdk import Choice, TypeSafeClient

MODEL = "jev-1.13.0"          # pin; jev-latest is an alias that moves
REPEATS = 15                  # matches the consistency cookbooks
PRICE_PER_MTOK_INPUT = 0.042  # output tokens are free
BUCKETS = [0.0, 0.5, 0.6, 0.7, 0.8, 0.9, 1.01]

QUESTIONS = {
    "department": Choice(
        instructions="Which team should handle this ticket?",
        criteria={
            "billing": "Charges, invoices, refunds, subscriptions",
            "technical": "Bugs, outages, integrations",
            "other": "Anything else",
        },
    ),
}

CASES = [
    {"id": "t1", "state": "I was charged twice for order A-104.", "label": "billing"},
    {"id": "t2", "state": "The export button 500s on Safari.", "label": "technical"},
    # ... your held-out cases
]


def evaluate() -> None:
    hits: list[tuple[float, bool]] = []
    per_case_labels: dict[str, Counter] = defaultdict(Counter)
    per_case_conf: dict[str, list[float]] = defaultdict(list)
    latencies: list[float] = []
    cost = 0.0
    models = Counter()

    with TypeSafeClient(timeout=30.0) as client:
        for case in CASES:
            for repeat in range(REPEATS):
                started = perf_counter()
                response = client.system_one(
                    state={"uid": f"{case['id']}:{repeat}", "ticket": case["state"]},
                    questions=QUESTIONS,
                    model=MODEL,
                )
                latencies.append(perf_counter() - started)
                models[response.model] += 1
                cost += (response.usage.input_tokens or 0) / 1e6 * PRICE_PER_MTOK_INPUT

                answer = response.answers["department"]
                per_case_labels[case["id"]][answer.choice] += 1
                per_case_conf[case["id"]].append(answer.confidence)
                if repeat == 0:  # score accuracy on the first draw only
                    hits.append((answer.confidence, answer.choice == case["label"]))

    # 1. Accuracy
    print(f"accuracy {sum(ok for _, ok in hits)}/{len(hits)}")

    # 2. Calibration: accuracy per confidence bucket -> read your threshold off this
    for low, high in zip(BUCKETS, BUCKETS[1:]):
        band = [ok for conf, ok in hits if low <= conf < high]
        if band:
            print(f"  confidence [{low:.2f},{high:.2f})  n={len(band):3d}  acc={sum(band) / len(band):.2%}")

    # 3. Repeatability: label flips and confidence spread per case
    for case_id, counts in per_case_labels.items():
        spread = statistics.pstdev(per_case_conf[case_id])
        flag = "  <-- FLIPS" if len(counts) > 1 else ""
        print(f"  {case_id}: {dict(counts)} conf_std={spread:.4f}{flag}")

    # 4/5. Latency and cost
    print(f"median latency {statistics.median(latencies) * 1000:.0f} ms over {len(latencies)} calls")
    print(f"total cost ${cost:.6f}   models answered: {dict(models)}")


if __name__ == "__main__":
    evaluate()

Cache results to a file keyed on case id, repeat index, rubric fingerprint, and model, so re-analysis costs nothing — that is what the cookbooks' JsonCache does (Cookbooks overview).

Before you publish the numbers

TypeSafe's public position is "run your own private evals, treat public ones with a grain of salt" (Blog: Lies, Damned Lies, and Benchmarks), and the company commits to no standard benchmark table, dated snapshots retired rather than hill-climbed, and publishing its own unflattering evidence.

The contract points the other way. MCA §2.3(f) prohibits customers from publishing "benchmarks or performance information about the Services" (Legal: MCA, DPA, privacy, data retention). Internal evaluation is exactly what the docs ask for; publishing your Jev numbers is a contractual matter — get permission first. This is a summary, not legal advice.

Related

Sources