Jev vs LAYA
LAYA is the open model that takes the Jev request verbatim and returns the Jev answer. Zero-shot it is not close; fine-tuned on the target suite it is level. The comparison is really three numbers: 0.343, 0.714, 0.726.
undisclosed · undisclosed · Proprietary (hosted API)
ModernBERT-large + decision head (RLCD) · 421M · Apache 2.0
Metatext decisions
Three hand-written workflows (support tickets, security incidents, agent traces), seven questions, about 123 judgments per model. Small, so treat a ±0.1 difference as a tie; every question has a majority baseline the model has to beat.
| Metric | Jev | LAYA |
|---|---|---|
| Decision accuracy | 0.863 | 0.520 |
| Questions over baseline | 6/7 | 3/7 |
| ECE raw → held-out fitted | 0.093 | 0.064 |
| p50 per case | 441 ms | 378 ms |
| Extraction F1 | n/a | n/a |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | LAYA |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.357 | 0.786 | 0.357 |
| agent_traces | needs_human | noul | 0.714 | 0.714 | 0.429 |
| customer_service | department | choice | 0.400 | 0.950 | 0.750 |
| customer_service | refund_requested | noul | 0.700 | 0.900 | 0.850 |
| customer_service | frustration | score | 0.500 | 0.950 | 0.400 |
| security_incidents | severity | choice | 0.312 | 0.875 | 0.562 |
| security_incidents | actively_exploited | noul | 0.688 | 0.938 | 0.438 |
typed-decisions
LAYA's public benchmark: four workflows × 100 JSON states × five typed questions = 2,000 decisions. Synthetic; gold is a teacher distribution and the label is its argmax. Large enough to carry an ordering.
| Metric | Jev | LAYA |
|---|---|---|
| Decision accuracy | 0.726 | 0.343 |
| Questions over baseline | 18/20 | 2/20 |
| ECE raw → held-out fitted | 0.079 → 0.049 | 0.196 → 0.020 |
| p50 per case | 405 ms | 3.2 s |
Per-question accuracy
| Task | Question | Type | Baseline | Jev | LAYA |
|---|---|---|---|---|---|
| td_agent_trace_observability | action | choice | 0.360 | 0.620 | 0.310 |
| td_agent_trace_observability | needs_review | noul | 0.550 | 0.600 | 0.580 |
| td_agent_trace_observability | outcome | choice | 0.500 | 0.800 | 0.400 |
| td_agent_trace_observability | risk | score | 0.340 | 0.630 | 0.340 |
| td_agent_trace_observability | urgency | score | 0.370 | 0.360 | 0.370 |
| td_customer_service | action | choice | 0.520 | 0.670 | 0.050 |
| td_customer_service | category | choice | 0.380 | 0.930 | 0.480 |
| td_customer_service | churn_risk | score | 0.470 | 0.860 | 0.380 |
| td_customer_service | needs_human | noul | 0.670 | 0.650 | 0.670 |
| td_customer_service | urgency | score | 0.420 | 0.840 | 0.300 |
| td_invoice_processing | discrepancy_severity | score | 0.470 | 0.570 | 0.010 |
| td_invoice_processing | disposition | choice | 0.450 | 0.650 | 0.440 |
| td_invoice_processing | duplicate | noul | 0.890 | 0.920 | 0.110 |
| td_invoice_processing | matches_order | noul | 0.540 | 0.960 | 0.490 |
| td_invoice_processing | urgency | score | 0.510 | 0.680 | 0.320 |
| td_security_incidents | credential_compromise | noul | 0.550 | 0.820 | 0.550 |
| td_security_incidents | disposition | choice | 0.740 | 0.790 | 0.050 |
| td_security_incidents | severity | score | 0.490 | 0.790 | 0.310 |
| td_security_incidents | true_positive | noul | 0.730 | 0.810 | 0.520 |
| td_security_incidents | urgency | score | 0.500 | 0.570 | 0.170 |
Verdict
Jev wins zero-shot by a wide margin on both suites. LAYA wins calibration out of the box (ECE 0.064 vs 0.093 on the Metatext suite) and matches Jev after a fine-tune on the suite (0.714 vs 0.726; 0.762 after temperature). Sell LAYA as "matches Jev after a fine-tune on your traffic", never as a zero-shot substitute.
Frequently asked questions
Is LAYA a drop-in replacement for Jev?
The API is: same state, same typed questions, same answer shape. The accuracy is not, until you fine-tune it on your decisions.
Related
Every open-weights alternative to TypeSafe's Jev, ranked by measured decision accuracy on the 2,000-decision typed-decisions suite: LAYA, Von, GLiNER2, GLiClass, OpenJEV, plus the Jev-shaped checkpoints not yet run.
The two open models that take the Jev request unchanged, on the same suites.
An independent benchmark of TypeSafe's Jev against every open System One alternative: decision accuracy, calibration (ECE) and latency on two suites, 2,123 judgments, same harness for every row.