$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/patterns

Confidence-gated routing

[ pattern ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ]#patterns · confidence · routing · safety · thresholds

TL;DR "The answer tells you what; confidence tells you whether to act." Put a global floor under everything (the example uses 0.6 → human), then give each action its own threshold scaled to its blast radius (the example uses > 0.85 before auto-approving a money transfer, and confirm-with-the-user otherwise).

Problem

A classifier gives you one answer and one number. Acting on the answer alone treats "almost certainly a balance check" and "probably an approval, maybe not" identically — even though one costs a wasted sentence and the other moves money.

From raw/docs/patterns__confidence-routing.md:

One of TypeSafe's most powerful features is confidence. By being intentional with the way you gate decisions on confidence, you can build systems that are both reliable and safe.

While you always want to have reasonable confidence in interpreting the user's intent, some actions are riskier than others and thus demand a higher confidence threshold.

Pattern

Use confidence as a second axis alongside the answer:

  1. A floor. Below it, nothing is automated — hand off to a person.
  2. Per-action thresholds above the floor. Each branch's threshold is set by the cost of acting on a misclassification, not by the model.
  3. A middle band for high-stakes actions. Between the floor and the action's threshold, confirm rather than act or escalate.

Confidence is reported on Choice and Score answers only; Noul returns a probability (noul) and no separate confidence. See Confidence vs probability.

Implementation

The documented example is a voice banking interface.

Step 1: determine the user's intent

Questions, verbatim from the source (the example sends no state; in a real request the transcribed utterance is the state — add "state": ... and "model": "jev-latest", see HTTP API: POST /v1/systemone and GET /v1/models):

{
  "intent": {
    "type": "choice",
    "instructions": "What action is the user requesting?",
    "criteria": {
      "check_balance": "Check the balance of an account",
      "approve_transfer": "Approve the pending transfer request",
      "other": "Something else"
    }
  }
}

Note the other option: a no-match outcome so the model is not forced to pick between two wrong answers.

Step 2: confidence-gated routing

action = response.answers["intent"]

# Below 0.6 confidence on any action, route to a human
if action.confidence < 0.6:
    route_to_support_agent(account_id)

elif action.choice == "check_balance":
    # Low stakes. 0.6 confidence is sufficient.
    show_balance(account_id)

elif action.choice == "approve_transfer":
    if action.confidence > 0.85:
        # High stakes, but high confidence. Safe to act automatically.
        approve_transfer(account_id)
    else:
        # High stakes, moderate confidence. Verify intent first.
        ask_user_to_confirm("Just to confirm: you would like to approve this transfer, is that correct?")

else:
    route_to_support_agent(account_id)

The same gate in TypeScript with @typesafe-ai/sdk 0.6.0 — adapted from the Python sample (not in upstream docs), same thresholds and branches. choice() builds the question and the answer carries choice, probabilities, and confidence; see JavaScript/TypeScript SDK: install, client, choice/score/noul.

import { choice, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY

const { answers } = await client.systemOne({
  state: transcript, // the transcribed utterance
  questions: {
    intent: choice("What action is the user requesting?", {
      check_balance: "Check the balance of an account",
      approve_transfer: "Approve the pending transfer request",
      other: "Something else",
    }),
  },
});

const action = answers.intent;

// Below 0.6 confidence on any action, route to a human
if (action.confidence < 0.6) {
  routeToSupportAgent(accountId);
} else if (action.choice === "check_balance") {
  // Low stakes. 0.6 confidence is sufficient.
  showBalance(accountId);
} else if (action.choice === "approve_transfer") {
  if (action.confidence > 0.85) {
    // High stakes, but high confidence. Safe to act automatically.
    approveTransfer(accountId);
  } else {
    // High stakes, moderate confidence. Verify intent first.
    askUserToConfirm("Just to confirm: you would like to approve this transfer, is that correct?");
  }
} else {
  routeToSupportAgent(accountId);
}

The source's reading of those numbers:

The 0.6 floor catches anything the model is genuinely uncertain about. Above that floor, each action type has its own threshold based on the consequences of acting on a wrong classification. Checking a balance at 0.6 is fine because the worst case is the user having to listen to the balance read-out. But approving a transfer requires very high confidence (>0.85), otherwise the system should ask the user to confirm.

0.6 and 0.85 are this example's constants, not model constants. Keep them in one place: the agent-skill guidance is "Put the constants (questions and thresholds) in a single place so they're easy to review."

When it fails

Variants

Related

Sources