Third partyLiteLLM · 2026-09-18
Measured on: 80 authored routing cases × 3 runs
- jev: Routing accuracy
- 95.0% (228/240) vs Claude Haiku 4.5 73.75%
- jev: Latency p50
- 127 ms vs 688 ms (5.4×)
- jev: Cost
- $0.0077 vs $0.1985 (96% cheaper)
Third-party, config given, cases not public.
Third partynibzard · 2026-09-18
Measured on: Banking77 (77-way), spam, code-word up to 512 options, permuted option order
- jev: Banking77 / spam / permuted
- 76.3% / 93.0% / 76.7%
- jev: ECE
- 0.246 (worst of the field; LLMs 0.039–0.122)
- jev: Option-order flip rate
- 13%
- jev: Latency p50
- 264–276 ms; 400 error at ≥ 256 options
- gpt-oss-120b: Banking77
- 81.3%
Third-party, full code, logs and frozen items public. The strongest independent calibration measurement of Jev.
Third partyickma2311 · 2026-09-18
Measured on: Banking77 (n=208), CLINC150 (n=200), pre-registered
- jev: Banking77
- 0.832 [0.779, 0.880] vs GPT-5.6 Terra 0.875; supervised bge-small + LR 0.933
- jev: CLINC150
- 0.870 vs Terra 0.915
- jev: Overconfidence
- confidence 1.0 on 102/200 items, 6 of them wrong
Third-party, pre-registered, code public.
Third partySimon Smith · 2026-09-18
Measured on: Banking77 full test (3,080), 24 BM25 few-shot examples in the state
- jev: Accuracy with examples in state
- 92.40% vs fine-tuned BERT 93.66%
- jev: Zero-shot screening (n=154)
- 79.22%
- jev: Cost / time
- $0.44, 6m51s for 3,080 cases
Third-party, code public.
Third partyakhilaaa3
Measured on: Two sets × 80 states / 293 questions, writer-labelled
- jev: Medium set
- 88.05%, ECE 0.042, $0.00017/state (Gemini 3.8 Flash 99.3%, Sonnet 5 98.6%)
- jev: Hard set
- 68.26%, ECE 0.089 (LLMs 89–95%)
Third-party, dataset public.
Third partyLargitData · 2026-09-20
Measured on: 100 multi-turn RAG-routing decisions (20 conversations × 5), multilingual
- Gemma 4 31B: All four sub-decisions correct
- 77.0%
- jev: All four sub-decisions correct
- 61.4% (p50 749 ms)
- laya-multilingual: All four sub-decisions correct
- 0.0% (p50 189 ms)
Third-party, per-item logs on GitHub.
Third partyLuni
Measured on: PhishNChips (2,000 emails) and typed-decisions
- jev: Phishing accuracy / ECE / AUROC
- 0.626 / 0.154 / 0.689 (Haiku 4.5 0.813)
- laya: Phishing accuracy
- 0.505 raw, 0.611 after Platt scaling
- laya-typed-decisions: typed-decisions
- 0.767 (ECE 0.212) vs Jev 0.727 (ECE 0.144)
Third-party scripts; the Jev rows are quoted from elsewhere, not measured here.
Third partyWotAI · 2026-09-18
Measured on: 150 prose-voice passages × 16 models, plus two more tasks
- jev: Accuracy / ECE / p50
- 66.0% / 0.121 / 455 ms
- Claude Sonnet 5: Accuracy / ECE / p50
- 71.3% / 0.062 / 1,674 ms
- jev: Accuracy in the 0.95–1.0 confidence bin
- 88.5% (n=122)
Third-party, data not public.
Third partyBen Greenberg · 2026-09-18
Measured on: 102 submissions × 3 runs
- jev: Accuracy (Choice + 4 Nouls)
- 100.0%; Choice alone 99.3%; Sonnet 5 99.0%
- jev: ECE
- 0.037 vs Sonnet 5 0.058
- jev: p50 / cost per 10k
- 378 ms / $2.27 vs 3,554 ms / $129.74
Third-party, private data.
Third partyLangChain · 2026-09-20
Measured on: 5 agent runs × 100 repetitions, human oracle
- jev: does_pass agreement
- 500/500; GPT-5.6 Terra 99.8%; Sonnet 4.6 80.0%
- jev: Run-to-run variance
- 92–913× lower than the LLM judges
- jev: Latency / cost
- 0.44 s, $0.00035 per call
Third-party, small n.
Third partynavinpai
Measured on: 400 intent, 400 MMLU-Pro, 100 relational questions
- jev: Intent / MMLU-Pro / relational
- 99.75% / 79.75% / 0.00%
- laya: Intent / MMLU-Pro / relational
- 63.0% / 13.5% / 8.0%
- Qwen3 4B (one pass): Intent / MMLU-Pro / relational
- 96.25% / 45.0% / 53.0%
Third-party; method described, code not confirmed.
Third partyJared Palmer
Measured on: decision-v4 and transfer-v4 frozen dev sets
- jev: In-distribution / out-of-domain
- 0.845 / 0.855
- kev-0.6b: In-distribution / out-of-domain
- 0.805 (ECE 0.078) / 0.598
Independent measurement of Jev via Vercel AI Gateway; repo public.
Third partyAman Kumar · 2026-09-18
Measured on: ~16k calls across four public sets and a private pipeline
- jev: Page-type gates
- 95.8–98.0% outcome agreement
- jev: Spam (n≈18.5k)
- 98.3% zero-shot ≈ TF-IDF logistic regression 98.4%
- jev: Phishing
- 62.6% vs Haiku 81.3%; 95% with five decomposed Nouls
Third-party, private data.
Third partyLangfuse (citing Good Start Labs) · 2026-09-18
Measured on: 6,003 rubric checks
- jev: Agreement with Claude Fable 5.1 verdict
- 91.5%
- jev: Cost per 1M graded
- $160 vs $33,000 (Fable) / $400 (Luna)
Secondary source.
Third partySam Reghenzi · 2026-09-21
Measured on: 50 products, taxonomy assignment
- jev: Mean latency
- 1.38 s vs 9.62 s agentic loop
- jev: LLM calls / tokens
- −56% calls, −36% tokens
- jev: Response time vs question count
- flat 0.32–0.39 s for 1–7 questions
Third-party, private data.
Third party0xNatoshi
Measured on: 237 replayed coding-agent turns
- jev: Simulated token spend
- ≈ −60% vs routing everything to the frontier model
- jev: Router overhead
- ~$0.00003 and 0.6 s per turn
Historical simulation; README says quota saved was not measured.
Third partyunverifiedArcher Hume · 2026-09-17
Measured on: ~10k probe calls
- jev: MMLU-Pro / ECE (claimed)
- 84.6% / 0.031
- jev: Throughput (claimed)
- ~30k tokens in ~160 ms; up to 1,500 questions per request
Page fetched; the figures came from a search snippet, not the visible body. Treat as unverified.