---
title: "Measurements, access routes and open replicas"
type: community
source_tier: community
tags: [measurements, benchmarks, access, replicas, community]
created: 2026-09-21
updated: 2026-09-21
confidence: medium
sources:
  - raw/community/archerhume-jevs-architecture-unmasked.md
  - raw/x/MiguelriosEN-2101029313906987422.md
  - raw/x/_aj-2102061534956662818.md
  - raw/x/LangChain-2101454284927959080.md
  - raw/x/0xidanlevin-2100937437325205568.md
  - raw/x/0xCarnagee-2101456270909690202.md
  - raw/x/0xMovez-2101325703635435523.md
  - raw/x/jerryjliu0-2101738281046294552.md
  - raw/x/startupideaspod-2101029801821941933.md
  - raw/x/nedwize-2100973868324417852.md
  - raw/x-repos/kyotofin__tax-doc-classifier.md
  - raw/x-repos/jerryjliu__docjev.md
  - raw/nate/jev-shaped-problems.md
  - raw/x/skeptrune-2101209390992994570.md
  - raw/x/swill1ams-2100421326389354624.md
  - raw/x-repos/r-ms__mini-jev.md
  - raw/x-repos/vinnylarouge__jevlike.md
  - raw/x-repos/TheoLeeCJ__SemIf.md
jev_version: "jev-1.13.0"
summary: "Numbers measured with Jev — cost, latency, calibration, token accounting, vendor harness figures — plus access routes and replica trust notes. Every figure is one run; re-measure yours."
---

# Measurements, access routes and open replicas

> **TL;DR** The numbers behind the verdicts in [[ideas/field-reports]]: per-call cost and latency people measured, throughput, accuracy on small sets, gateway routes, and trust notes on open replicas. Every figure is one person's run on their own workload; re-measure before relying on any of it ([[guides/testing-and-evaluation]]).

## Measured numbers

Never quote these as facts. Each is one poster's — or one vendor's own — run.

| Number | Reported by | Setup | Caveat |
|---|---|---|---|
| $0.00035/call, 0.44 s avg; $0.34 total vs $28.17 for Claude Sonnet 4.6; mean per-case variance 0.0000149 | @LangChain | Deep Agents weather agent, 5 fixed traces, 100 reps/judge, human oracle, `langchain-typesafe==0.0.1a2` | 5 cases; Jev service version not recorded |
| 112x lower model cost than GPT-6 Astra (code-execution computer use); 245x vs screenshot computer use; WebMCP cut cost a further 18% | @0xidanlevin | WindTunnel, 49 tasks × 8 sites, Jev + Mercury 2.5 | Open, reproducible; compares harnesses, not models |
| 1,284 tool calls in one session, 938 yes/no; bill $64.77 → $26.16; $0.000041/decision; 38 ms vs 1.84 s | [@0xCarnagee](https://x.com/0xCarnagee/status/2101456270909690202) | One session, replayed side by side | Promotional, single run |
| 100,000 X posts in 20.4 s for $0.67, 14 Nouls each; Claude Opus 5 did 214 posts for $0.98 on the same clock | [@0xMovez](https://x.com/0xMovez/status/2101325703635435523) | Self-built analyser, one parallel pass | Opus leg is truncated and extrapolated to $458, not equal work |
| Classify 138.6 ms vs 794.3 ms median (5.73x); split 209.6 ms vs 1,352.3 ms (6.45x), Jev 1.13.0 vs GPT-5.6 Luna, decision time only | [@jerryjliu0](https://x.com/jerryjliu0/status/2101738281046294552), DocJev README | 40 public-finance PDFs, 8 packets, text pre-extracted by LiteParse | Small pilot; OCR time excluded from these medians; see [[ideas/patterns-data]] P26 |
| 1,700 real emails, ~4M input tokens, 18¢; ~200 ms/query; 4 questions each (category, priority, spam, reply) | Ryan Vogel via @startupideaspod | personal inbox, one pass | Arithmetic consistent with $0.042/Mtok (4M ≈ $0.17); latency self-reported |
| 50 calls at 25 concurrent: 741 ms, no 429s; all 50 at once was slower (~1 s). Claim check on 16 labelled headlines: true 0.83-0.90, false 0.01-0.12 | [Nate B. Jones](https://unlock-ai.natebjones.com/guides/jev-shaped-problems) | run through OpenRouter | One run; 25-way concurrency is a starting point, not a limit |
| $0.00115 vs $0.039 per page and ~0.5 s vs ~3.3 s against a Claude Sonnet classifier; 0 wrong on 1,067 tax-form pages, 38 under a 0.95 gate | [@nedwize](https://x.com/nedwize/status/2100973868324417852), `kyotofin/tax-doc-classifier` | open repo, reproducible eval | His corpus; blank and benchmark forms, not messy scans |
| ~30k tokens in ~160 ms; server time barely moved up to ~100 questions, then rose steadily; 1,500 questions still returned in a few hundred ms; a 200-option question was as quick as a 2-option one | [Archer Hume](https://archerhume.com/posts/jevs-architecture-unmasked), `jev-1.13.0`, 2026-09-17 | 192 + 148 sequential requests, shuffled, 8 repeats per size | Times are the `x-envoy-upstream-service-time` header: shared service, queueing included, not a hardware benchmark |
| Token accounting is additive: 268 input tokens for one minimal Noul, 276 for two, 318 for a Noul + 2-option Choice + 2-level Score; a 23k-token state with 5,000 questions was accepted | same | 311 accounting + 35 limit-boundary requests | `verified` against the 64k-per-request / 32k-per-branch budgets with `state` counted once ([[reference/models-and-pricing]]) |
| Calibration: 10-bin ECE **0.031** over 1,200 MMLU items (990 of them in the 0.9–1.0 bin); MMLU-Pro 84.6%; fresh 3-digit multiplication 86.7% correct at mean top probability 0.83; two-step word problems 32% at 0.30; modular exponentiation 56% at 0.35 | same | 6,800 benchmark records + 190 generated maths items | Recomputed from recorded probabilities, not the API `confidence` field. Family averages can hide bin-level error; maths stays a documented weakness ([[concepts/jaggedness-jev-1-13]] §2) |
| Option-set effects: reversing a ticket's options moved the top probability 0.84–0.89 → 0.93–0.96; adding one irrelevant option moved the log-odds between two existing options +0.38 → +0.11 (mean −0.28, paired 95% ≈ −0.36…−0.19, lower in all 10 blocks) | same | 181 order + 96 reference-card + 50 block-randomised requests | `unverified` against docs — one scenario per template; advisory reading in [[ideas/field-reports]] |
| `output_tokens` is billing arithmetic, not decoding: a Noul request costs 4 shared tokens + 15 per answer + the token length of each question id (which the docs say the model never sees); a 255-option response reported 2,714 | same | 311 requests | Output tokens are free ([[reference/models-and-pricing]]); never divide them by request duration and call it speed |
| Jev's tokenizer matches none of 192 public tokenizers across 415 probes (closest Qwen at 348/415); every digit is a separate token; 8 `a`s count as 1 token, 16 as 4 | same | 445 requests | `unverified`. Local `tiktoken` or character-count estimates drift in both directions — leave headroom under the caps |
| 100,000 compliance alerts: **>$290K** on Opus 5 vs **<$26K** through the harness; 50–95% saving claimed for workflows vs a frontier-model agent generally; ~$3 per alert for a frontier agent ($2.89 on their production agent → **$0.25** per alert through the tuned workflow) | Grep.ai's AgentRun, [@MiguelriosEN](https://x.com/MiguelriosEN/status/2101029313906987422) and [@_aj](https://x.com/_aj/status/2102061534956662818) | one AML alert-review job: 100-alert arm comparison, then a 1,000-alert tuning run | **The vendor's own figures about its own product**, one regulated domain. Pattern: [[ideas/patterns-agents]] P37 |
| By the end of the run **98 of every 100** cases went through the workflow (hand-back ~1 in 70 over the last 300); their compliance judge scored 90% on the first 100 full-agent alerts and 95% over the last 300; one alert fell from 826 tool calls / 51 min to ~30 calls / ~3 min; ~30 Jev questions per alert cost about a third of a cent; 868 hits re-judged by replay in 2 minutes for a few cents; the tuning run cost $1,049 | same | same | Same caveat. The accuracy grader is their own LLM judge against historical labels, not a fresh human relabel |
| "About 150 ms" per typed answer, at a few hundred-thousandths of a dollar | same | vendor description | Consistent with $0.042/Mtok and the 70–500 ms band others report; not independently measured |

TypeSafe's own 193.6x / 444.6x is a *workflow-average ceiling* ([[concepts/workflow-evals]]). @cyrilXBT calls them "a ceiling, not a typical result": the multiplier comes from a workflow's decision-to-generation ratio, not a head-to-head call.

## Open replicas and Jev-like alternatives

What each one is and how it works: [[ideas/repos-apps]]. Trust notes only here.

- None reproduces Jev's model. SemIf (ex-openjev) says so itself, and scores its Jev side from TypeSafe's published records (102 aligned rows), not a live endpoint.
- Calibration is the gap: [r-ms/mini-jev](https://github.com/r-ms/mini-jev) states its shares are normalized scores, **not calibrated probabilities**; [@skeptrune](https://x.com/skeptrune/status/2101209390992994570) calls his DeepSeek-based endpoint "not as good as jev".
- Task-trained tiny models can win narrow tasks (NanoJev above) and lose badly elsewhere (`vinnylarouge/jevlike`'s chess checkpoint lost 48/50 to Stockfish level 0).
- "5x faster on-device" ([@swill1ams](https://x.com/swill1ams/status/2100421326389354624)) and "already above Jev" ([@0xCVYH](https://x.com/0xCVYH/status/2101171688683585622)) come with no methodology: `unverified`.

**For a build decision.** The replicas show the *interface* is commodity — one forward pass, read scores over a closed option set — so the API shape is not lock-in and porting later is realistic. They do not show that the *calibration* is commodity: mini-Jev says outright its scores are not calibrated probabilities, and [[patterns/confidence-routing]] rests on calibration ([[concepts/confidence]]). So prototype on the hosted API, keep questions and thresholds in your own code, and self-host only once a labelled set proves a local model matches on *your* decisions. Replica claims are unverified too.

## Access, waitlist and practicalities

All `unverified`; confirm against [[reference/models-and-pricing]] and [[entities/typesafe-console]] before relying on any of it.

- Official waitlist approved in roughly a day ([@studio_yebisu](https://x.com/studio_yebisu/status/2100686990090047569), 2026-09-17).
- Reachable without the waitlist via Vercel AI Gateway as model `typesafe-ai/jev` ([@ai_agent_dev](https://x.com/ai_agent_dev/status/2100529814289678514)), and via OpenRouter from 2026-09-18 ([@studio_yebisu](https://x.com/studio_yebisu/status/2101065176069886152)). First-hand: [Nate B. Jones](https://unlock-ai.natebjones.com/guides/jev-shaped-problems) ran all his tests through `POST https://openrouter.ai/api/alpha/decisions` with model `typesafe/jev-1.13` and the same `state`/`questions` body; alpha endpoint, may change; not in TypeSafe's docs. Gateways add their own pricing, latency and data terms.
- A third-party `langchain-typesafe==0.0.1a2` package exists (@LangChain). Not a TypeSafe SDK; see [[reference/python-sdk]].
- **Don't assume bit-exact repeats.** Archer Hume's probes found small differences between identical requests, and between duplicate questions inside one request, plus varying response key order. Measure on distributions, not on equality; don't key a cache or a golden test on exact probabilities.
- **Measure latency the way the study did** if you want comparable numbers: server-reported upstream time (`x-envoy-upstream-service-time`), requests issued one at a time in shuffled order, several repeats per size. Local wall-clock adds your network and client.
- **Legal:** the MCA of 2026-08-27 prohibited publishing benchmarks (§2.3(f)); the MCA dated 2026-09-19 dropped that clause ([[reference/legal-and-data]]). Numbers here published before 2026-09-19 were published under the old terms. Check the current MCA before publishing your own.

## Related

- [[ideas/field-reports]] — what held up, what broke, and what it means when advising
- [[ideas/repos-apps]] — what each replica is and how it works
- [[concepts/workflow-evals]] — where TypeSafe's own 193.6x / 444.6x comes from
- [[reference/models-and-pricing]], [[entities/typesafe-console]] — official price, limits and access

## Sources

Links are inline in each row. Raw captures (`raw/x/`, `raw/x-repos/`, `raw/nate/`, `raw/community/`) live in the private repo.