System One scorecard

The scorecard is the Metatext suite: three workflows, seven questions, 123 judgments, plus an eight-case extraction task. It is small, so a ±0.1 gap is a tie, but it is the one place every checkpoint we have ever run appears on one table, including the GLiClass payload variants.

Leads on decision accuracy: Jev — 0.863 on Metatext decisions

#	Model	Decisions	Beats baseline	ECE raw → held-out fitted	p50 / case	Extraction F1
1	JevTypeSafehosted	0.863	6/7	0.093	441 ms†	n/a
2	GLiNER2 largeFastino	0.716	6/7	0.261	627 ms†	0.968
3	GLiNER 2.5 baseFastino	0.643	5/7	0.251	332 ms†	0.968
4	GLiClass large v3.0Knowledgator correct payload, no prompt	0.642	5/7	0.250	977 ms	n/a
5	GLiNER2 baseFastino	0.631	3/7	0.350	190 ms†	0.935
6	GLiClass multilang ultraKnowledgator criteria + single-hypothesis Noul (correct payload)	0.628	4/7	0.185	672 ms	n/a
7	GLiClass base v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.606	5/7	0.146	300 ms	n/a
8	GLiClass modern large v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.602	4/7	0.234	572 ms	n/a
9	GLiNER 2.5 smallFastino	0.574	4/7	0.172	132 ms†	0.812
10	GLiClass instruct base v1.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.559	3/7	0.390	323 ms	n/a
11	GLiClass instruct large v1.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.545	3/7	0.308	1.0 s	n/a
12	GLiGuard 300MFastino	0.537	3/7	0.230	155 ms†	0.000
13	LAYAConvai Innovations	0.520	3/7	0.064	378 ms†	n/a
14	Vonwfzyx	0.509	4/7	0.234→ 0.122	597 ms	n/a
15	GLiNER 2.5 multiFastino	0.494	4/7	0.244	346 ms†	0.885
16	LAYA typed-decisionsConvai Innovations	0.475	3/7	0.131	375 ms†	n/a
17	GLiNER2 multiFastino	0.464	2/7	0.354	202 ms†	1.000
18	LAYA multilingualConvai Innovations	0.431	2/7	0.284	149 ms†	n/a
19	OpenJEVopen-jev	0.427	3/7	0.210	50 s†	n/a
20	GLiClass edge v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.418	2/7	0.287	41 ms	n/a
21	GLiNER multitask v1.0Knowledgator class names as labels	0.378	1/7	0.223	1.8 s	n/a

Ranked by decision accuracy. Metatext decisions: Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat. Every model is zero-shot on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.

Same ranking on the other suite: typed-decisions.

Frequently asked questions

Why two suites?

The Metatext suite is hand-written and small; it was the first exam and it is the only one with an extraction task. typed-decisions is public and twenty times larger, so it carries the ranking. A model that wins one and not the other is worth a second look.

What does "beats baseline" mean?

For every question the majority class is a free baseline. A model that does not beat it on a question has learned nothing useful about that question. The column counts questions won.

System One model benchmark

The benchmark for System One models: decision models that answer typed questions with calibrated probabilities. Jev and every open alternative on the same 2,000-decision suite, with accuracy, wins over the majority baseline, ECE and latency.

Jev benchmark

An independent benchmark of TypeSafe's Jev against every open System One alternative: decision accuracy, calibration (ECE) and latency on two suites, 2,123 judgments, same harness for every row.

Extraction benchmark

Micro-F1 on an eight-case entity extraction task for the models that have a span head. Jev, LAYA, Von and GLiClass never return spans; only the GLiNER family does.

Fastest System One model

System One models ranked by p50 latency per three-question case on a four-core CPU, against Jev's hosted round trip.

Frequently asked questions

Related