Cookbook: Hierarchical classification

TL;DR One Choice per sibling set, with criteria mapping opaque keys (c0, c1, …) to the child labels; the answer's probabilities are the edges. Score a path by product(edge_probabilities) ** (1 / decisions) and keep the best K = 3 — beam search matched 4 of 4 expected leaves where greedy matched 2 of 4.

Goal

Classify a document into a deep hierarchy by traversing it node by node to the correct leaf. Greedy search takes the highest-probability child at every node and cannot recover from one early mistake; beam search keeps K plausible paths, classifies every frontier in parallel, and lets deeper evidence repair an ambiguous early decision.

The cookbook argues that decomposing classification into a hierarchy buys observability (which nodes your misclassifications occur in; how often each node and edge is traversed) and testability (unit-test hierarchy updates and measure their impact).

Inputs / state shape

The state is the plain document string. The hierarchy is a nested dict[str, Tree] of direct-child menus, loaded from four pinned sources:

MeSH is a DAG — one descriptor can appear under multiple parents — so the demo expands its official tree-number paths. The CookSafe listing is a frozen snapshot rather than a live walk, because line order is significant: sibling options are asked in the order they appear, so it is part of the question, not presentation.

The four documents and their expected leaves, verbatim:

Questions asked

Exactly one question type, asked once per sibling set. Verbatim:

def child_question(labels: tuple[str, ...]) -> tuple[Choice, dict[str, str]]:
    """Build the direct-child Choice and its reversible option mapping."""
    keys = {f"c{i}": label for i, label in enumerate(labels)}
    question = Choice(
        instructions="Which direct child category best matches this document?",
        criteria=keys,
    )
    return question, keys

• type: Choice
• instructions: Which direct child category best matches this document?
• criteria: {"c0": "<label 0>", "c1": "<label 1>", ...} — the option keys are synthetic (c0, c1, …) and the child labels are the option descriptions, so the mapping is reversible and label text never has to be a valid key.

The same instructions string is reused at every depth; all the node-specific information lives in criteria.

Combining logic in code

import os
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

from typesafe_sdk import Choice, RetryPolicy, TypeSafeClient

HIERARCHIES = load_hierarchies()
MODEL, BEAM_WIDTH, MAX_DEPTH, EPSILON = "jev-1.12", 3, 12, 1e-9
client = TypeSafeClient(
    api_key=os.environ["TYPESAFE_API_KEY"],
    retry=RetryPolicy(max_retries=5, backoff_initial=1.0, backoff_max=20.0),
)


def choose(state: str, labels: tuple[str, ...]) -> dict[str, float]:
    """Ask one atomic direct-child question and return its distribution."""
    if len(labels) == 1:
        return {labels[0]: 1.0}
    question, keys = child_question(labels)
    response = client.system_one(
        state=state, questions={"child": question}, model=MODEL
    )
    probabilities = response.answers["child"].probabilities
    return {label: probabilities[key] for key, label in keys.items()}


def extend_candidate(candidate: dict, label: str, probabilities: dict[str, float]) -> dict:
    """Append one edge and recompute its geometric-mean path score."""
    is_decision: bool = len(probabilities) > 1
    probability_product: float = candidate["probability_product"] * (
        max(probabilities[label], EPSILON) if is_decision else 1.0
    )
    decision_count: int = candidate["decision_count"] + is_decision
    return {
        "path": candidate["path"] + (label,),
        "probability_product": probability_product,
        "decision_count": decision_count,
        "score": probability_product ** (1 / decision_count) if decision_count else 1.0,
    }

The beam itself, trimmed of diagram bookkeeping:

def beam_search(hierarchy) -> dict:
    """Parallel width-three beam search using geometric-mean probability."""
    beam = [{"path": (), "probability_product": 1.0, "decision_count": 0, "score": 1.0}]
    for _ in range(MAX_DEPTH):
        expandable = [c for c in beam if subtree(hierarchy.tree, c["path"])]
        finished = [c for c in beam if not subtree(hierarchy.tree, c["path"])]
        if not expandable:
            break
        with ThreadPoolExecutor(max_workers=BEAM_WIDTH) as executor:
            distributions = list(
                executor.map(
                    lambda c: choose(
                        hierarchy.document, tuple(subtree(hierarchy.tree, c["path"]))
                    ),
                    expandable,
                )
            )
        expanded = []
        for candidate, probabilities in zip(expandable, distributions, strict=True):
            for label in probabilities:
                expanded.append(extend_candidate(candidate, label, probabilities))
        beam = sorted(finished + expanded, key=lambda c: c["score"], reverse=True)[:BEAM_WIDTH]
    return {"beam": sorted(beam, key=lambda c: c["score"], reverse=True)}

Greedy is the same walk with label = max(probabilities, key=probabilities.get) and no beam.

Metrics.

• path_score = product(edge_probabilities) ** (1 / decisions) — used for pruning and comparing paths, length-normalized so shallow and deep leaves compare fairly.
• separation = top_path_score / second_path_score — reported but not used for pruning. Near 1× is ambiguous; a large ratio means clear separation.
• Alternatives the cookbook names: min(top_prob/second_top_prob) optimizes for paths with very clear decisions at every node; exp(mean(log(probs))) avoids precision errors for hierarchies deeper than ~10 layers.

Note that a sibling set of size 1 is not a decision: choose short-circuits to {label: 1.0} with no API call, and decision_count is not incremented.

Results / what the cookbook reports

With BEAM_WIDTH = 3, MAX_DEPTH = 12, model jev-1.12: beam search matched 4 of 4 expected leaves; greedy search matched 2 of 4. Keeping three paths recovered the expected classification for CPC patents and Shopify products.

Four examples is the whole evaluation — treat this as a demonstration, not a benchmark (inferred).

Adapting it to a new domain

1. Produce a nested dict of direct-child menus for your taxonomy (filesystem, org chart, moderation policy, LLM skill catalog, ontology). Fix the sibling order and keep it fixed, since order is part of the question.
2. Leave child_question alone unless your labels need more than a name — if a label is cryptic, put a description in criteria instead of the bare label.
3. Tune BEAM_WIDTH (cost per level scales with it, latency barely does because frontiers run in parallel) and MAX_DEPTH to your deepest path.
4. Log separation_ratio per document and route low-separation documents to review.

Gotchas

• Choice option limits. Very wide sibling sets will hit the Choice option ceiling (255 options, per Cookbook: Pre-parsed value extraction and Cookbook: Line-by-line search); split a wide menu into chunks and run the same procedure over the winners (inferred for this cookbook, stated in the others).
• Deep trees lose precision. The cookbook explicitly recommends exp(mean(log(probs))) past ~10 layers; EPSILON = 1e-9 floors any zero edge so the product never collapses.
• A single-child level is free but invisible — it contributes no decision and no probability.
• MeSH is a DAG, so the same descriptor appears on several paths; the demo expands tree numbers into distinct paths rather than deduplicating.
• Retries matter at this call volume. The client is constructed with RetryPolicy(max_retries=5, backoff_initial=1.0, backoff_max=20.0).
• Install line. This cookbook prints no pip install line; it imports from cooksafe import JsonCache and reads os.environ["TYPESAFE_API_KEY"]. Sibling cookbooks use pip install ... "typesafe-sdk>=0.5.7" cooksafe --extra-index-url https://pypi.typesafe.ai/. cooksafe lives on TypeSafe's private index and pypi.typesafe.ai returned 404 publicly as of 2026-09-17, so use pip install typesafe-sdk and reimplement JsonCache — a decorator that memoizes a function's return value into a JSON file keyed on its arguments, so re-running replays cached answers with no API spend.

Related

• Choice questions — probabilities over a fixed option set
• Structured instructions, options, levels, criteria — structured instructions, options, levels, criteria
• Speculative fan-out — running every frontier's question concurrently
• Intent routing — the shallow, one-level version of this
• Cookbook: Classification using confidence — reporting a parent class when the child is uncertain
• Cookbook: Skill suggestion — rank-then-re-read as an alternative to beam search
• Cookbooks overview — the full cookbook catalog

Sources

• raw/docs/cookbooks__hierarchical_classification.md (https://docs.typesafe.ai/cookbooks/hierarchical_classification.md)