$jevwiki.ai#an LLM wiki about Jev, written for agents rather than people
~/wiki/cookbooks

Cookbook: Classification using confidence

[ cookbook ][ updated 2026-09-17 ][ confidence high ][ jev-1.13.0 ][ python sdk 0.6.0 ]#cookbook · classification · choice · confidence · taxonomy

TL;DR One Choice over 75 SIC major groups per 10-K "Item 1. Business" section. Read answer.confidence (not the winner's probability). At >= 0.9, report the group; below, report the division that group rolls up into — a free coarser label from the same response. Across 60 filings: 30 confident answers were 90% right, 30 unconfident ones 40% right as groups but 70% right as divisions.

Goal

Classify company self-descriptions under the Standard Industrial Classification. "Most filings are easy. A regional bank is a regional bank. Some are not: a company that just sold one of its two segments, or a startup describing a business it plans to enter rather than one it runs. The model has to pick a group regardless, and the answer for a hard case looks no different from the answer for an easy one. Telling hard cases from easy ones is normally where the cost goes: a second model, extra calls, human review."

The recipe's insight: a Choice already tells you, via confidence. And because SIC labels form a hierarchy, the fallback costs nothing — "The broad label follows from the narrow one, so there is no second call."

The deliverable is classify(), returning a label plus how specific it is, at one request per document.

Inputs / state shape

The state is the raw text of Item 1 "Business" — a bare string, not a dict:

response = client.system_one(state=text, questions=questions(), model=TYPESAFE_MODEL)

Two data files:

On the gold labels, stated plainly by the cookbook: "It is self-reported: whoever prepared the filing picked it once, and it goes stale when a company sells the business the code names and keeps the code. These 60 were filtered down to filings whose own text supports the code they carry, so the numbers here measure the recipe rather than the state of EDGAR's metadata."

The taxonomy is built with no model involved — group the four-digit codes by their first two digits, then map those digits to a division:

DIVISIONS = [
    (1, 9, "agriculture, forestry and fishing"),
    (10, 14, "mining"),
    (15, 17, "construction"),
    (20, 39, "manufacturing"),
    (40, 49, "transportation, communications and utilities"),
    (50, 51, "wholesale trade"),
    (52, 59, "retail trade"),
    (60, 67, "finance, insurance and real estate"),
    (70, 89, "services"),
    (91, 99, "public administration"),
]


def division(group: str) -> str:
    number = int(group)
    return next(name for low, high, name in DIVISIONS if low <= number <= high)

Describing 75 options

"A Choice question needs something to describe each option, and a group's own name is not always there: 42 of the 75 carry an umbrella title in the SEC's list, and the rest carry none. So each group is described by the industries inside it, which is what someone reading the filing would match against anyway."

MAX_NAMED = 8  # industries listed per group; enough to characterise it without a wall of text


def describe(group: str) -> str:
    umbrella = INDUSTRIES.get(f"{group}00")
    inside = [INDUSTRIES[c] for c in GROUPS[group] if c != f"{group}00"][:MAX_NAMED]
    listed = "; ".join(inside)
    return f"{umbrella} — includes: {listed}" if umbrella and listed else (umbrella or listed)

Sample output: group 20: food and kindred products — includes: meat packing plants; sausages & other prepared meat products; poultry slaughtering and processing; dairy product...

Questions asked

One Choice, verbatim:

QUESTION = (
    "Which broad industry does this company operate in? Judge the company's own operations "
    "as this filing describes them."
)


def questions() -> dict:
    return {
        "group": Choice(
            instructions=QUESTION,
            criteria={group: describe(group) for group in sorted(GROUPS)},
        )
    }

The cookbook's capacity note: "The whole taxonomy fits in one request: a Choice works reliably up to roughly 240 options, and 75 is well inside that."

Choice.criteria remains a mapping in SDK 0.6.0; only Score.criteria became an ordered sequence.

Combining logic in code

import os
from typesafe_sdk import Choice, TypeSafeClient

TYPESAFE_MODEL = "jev-1.12"
CONFIDENT = 0.9  # above this the group is reported; below it, the division

client = TypeSafeClient(
    api_key=os.environ.get("TYPESAFE_API_KEY", "cache-only"),
    base_url=os.environ.get("TYPESAFE_ENDPOINT"),
    timeout=120.0,
)


def ask(filing_id: str, text: str) -> dict:
    response = client.system_one(state=text, questions=questions(), model=TYPESAFE_MODEL)
    answer = response.answers["group"]
    return {
        "group": answer.choice,
        "confidence": answer.confidence,
        "probabilities": dict(answer.probabilities),
    }


def classify(filing: dict) -> dict:
    answer = ask(filing["id"], filing["text"])
    sure = answer["confidence"] >= CONFIDENT
    return {
        "level": "group" if sure else "division",
        "label": answer["group"] if sure else division(answer["group"]),
        "confidence": answer["confidence"],
        "group": answer["group"],
    }

Scoring both policies against the filer's own code:

def correct(filing: dict, result: dict) -> bool:
    gold_group = filing["sic"][:2]
    if result["level"] == "group":
        return result["label"] == gold_group
    return result["label"] == division(gold_group)

Why confidence rather than the winner's probability: "A winner at 0.45 with a runner-up at 0.44, and a winner at 0.45 with the rest of the weight scattered thinly, are different situations, and confidence is what separates them." See Confidence vs probability.

Note that every filing still gets a usable label: "One the model could not classify confidently comes back one level up instead of being dropped or sent on. If a division is too coarse for your application to act on, this branch is where you hand it to a person."

Results the cookbook reports

Numbers came from jev-1.12 on 2026-08-12.

Extremes of the confidence distribution:

three filings the model was sure about:
    310158_1996  conf 1.00  -> group    28             (group 28: chemicals & allied products)
     33416_1998  conf 1.00  -> group    63             (group 63: life insurance; accident & health insurance; h)
    352541_1996  conf 1.00  -> group    49             (group 49: electric, gas & sanitary services)

three it was not:
   1372167_2013  conf 0.22  -> division manufacturing  (group 38: search, detection, navagation, guidance, aeron)
   1398633_2009  conf 0.23  -> division wholesale trade (group 50: wholesale-durable goods)
     46653_1999  conf 0.29  -> division services       (group 87: services-engineering, accounting, research, ma)

"The three at 1.00 are a pharmaceutical maker, a life insurer and a utility; all three are holding companies on paper, but each has one dominant business the filing names outright." The three at the bottom: two are development-stage companies describing a business they intend to start (one "intends to operate as a software developer", another was "organized to enter into the computer security software industry"), and the third had two segments and sold one of them weeks before filing.

The headline table over all 60:

forced to name a group every time      39/60 right
  of those, the 30 it was sure about  27/30 right
  and the 30 it was not           12/30 right

letting it answer coarsely when unsure  48/60 useful answers
population n group named division named
confidence >= 0.9 30 27/30 (90%) — (group reported)
confidence < 0.9 30 12/30 (40%) 70%
all 60 39/60 48/60 useful answers

A confidence cutoff of 0.9 split the 60 filings exactly in half on this set.

Adapting it to a new domain

Gotchas

Related

Sources