Best-calibrated System One model

Untuned ECE on the Metatext suite. LAYA ships the best-calibrated open probabilities; GLiNER2 and GLiClass are over-confident out of the box and need one fitted temperature.

Leads on ece (lower is better): LAYA — 0.064 on Metatext decisions

#	Model	Decisions	Beats baseline	ECE raw → held-out fitted	p50 / case	Extraction F1
1	LAYAConvai Innovations	0.520	3/7	0.064	378 ms†	n/a
2	JevTypeSafehosted	0.863	6/7	0.093	441 ms†	n/a
3	LAYA typed-decisionsConvai Innovations	0.475	3/7	0.131	375 ms†	n/a
4	GLiClass base v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.606	5/7	0.146	300 ms	n/a
5	GLiNER 2.5 smallFastino	0.574	4/7	0.172	132 ms†	0.812
6	GLiClass multilang ultraKnowledgator criteria + single-hypothesis Noul (correct payload)	0.628	4/7	0.185	672 ms	n/a
7	OpenJEVopen-jev	0.427	3/7	0.210	50 s†	n/a
8	GLiNER multitask v1.0Knowledgator class names as labels	0.378	1/7	0.223	1.8 s	n/a
9	GLiGuard 300MFastino	0.537	3/7	0.230	155 ms†	0.000
10	GLiClass modern large v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.602	4/7	0.234	572 ms	n/a
11	Vonwfzyx	0.509	4/7	0.234→ 0.122	597 ms	n/a
12	GLiNER 2.5 multiFastino	0.494	4/7	0.244	346 ms†	0.885
13	GLiClass large v3.0Knowledgator correct payload, no prompt	0.642	5/7	0.250	977 ms	n/a
14	GLiNER 2.5 baseFastino	0.643	5/7	0.251	332 ms†	0.968
15	GLiNER2 largeFastino	0.716	6/7	0.261	627 ms†	0.968
16	LAYA multilingualConvai Innovations	0.431	2/7	0.284	149 ms†	n/a
17	GLiClass edge v3.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.418	2/7	0.287	41 ms	n/a
18	GLiClass instruct large v1.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.545	3/7	0.308	1.0 s	n/a
19	GLiNER2 baseFastino	0.631	3/7	0.350	190 ms†	0.935
20	GLiNER2 multiFastino	0.464	2/7	0.354	202 ms†	1.000
21	GLiClass instruct base v1.0Knowledgator criteria + single-hypothesis Noul (correct payload)	0.559	3/7	0.390	323 ms	n/a

Ranked by ece (lower is better). Metatext decisions: Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat. Every model is zero-shot on this suite. Decisions is the task-weighted mean accuracy of the best configuration per model; ECE is the mean expected calibration error over tasks, and the arrow shows the held-out ECE after one temperature per primitive fitted on the other half. Latency is p50 per three-question customer-service case on four CPU cores (Jev: API round trip); † means taken from the benchmark run rather than a clean timing pass. Updated September 2026.

Same ranking on the other suite: typed-decisions.

Frequently asked questions

Which open model is best calibrated?

LAYA (ECE 0.064 on the Metatext suite), because it ships a fitted temperature per primitive. Every GLiNER2 and GLiClass row is above 0.15 untuned.

Jev calibration

Expected calibration error of Jev and every open System One model, raw and after one fitted temperature per primitive judged on a held-out fold. TypeSafe publishes no calibration number; this page does.

System One scorecard

One table, every System One model: Jev, LAYA (three checkpoints), Von, OpenJEV, seven GLiNER2 and seven GLiClass checkpoints. Decision accuracy, calibration and latency on the Metatext suite, extraction F1 where the model has a span head.

Frequently asked questions

Related