Third partyLuni
laya-jev-benchmark
Measured on: PhishNChips (2,000 emails) and typed-decisions
- Phishing accuracy
- 0.505 raw, 0.611 after Platt scaling
Third-party scripts; the Jev rows are quoted from elsewhere, not measured here.
LAYA
Convai InnovationsOpen weights
The open-weights System One: takes the Jev request verbatim and returns the Jev answer. One forward pass scores every option of every question; ships with a fitted temperature per primitive.
- typed-decisions
- 0.343
- Metatext suite
- 0.520
- ECE untuned
- 0.196
- p50 per case
- 378 ms
2/20 over baseline
3/7 over baseline
→ 0.020 held-out fitted
four CPU cores
Specifications
- Checkpoint
- convaiinnovations/laya
- Publisher
- Convai Innovations
- Mechanism
- Encoder with a decision head (System One)
- Backbone
- ModernBERT-large + decision head (RLCD)
- Parameters
- 421M
- License
- Apache 2.0
- Runs as
- open weights, self-hosted
- Library
- laya
- Primitives
- Choice, Score, Noul
- Pricing
- none — your compute
Metatext decisionsrank 13 of 21
Full leaderboard →| Configuration | Mode | Decisions | Beats baseline | ECE |
|---|---|---|---|---|
| Noul phrased as a question (ablation)ablation | systemone | 0.556 | 3/7 | 0.126 |
| default payloadheadline | systemone | 0.520 | 3/7 | 0.064 |
Per-question accuracy — default payload, systemone mode, 120 judgments
| Task | Question | Type | Accuracy | Baseline | Beats? |
|---|---|---|---|---|---|
| agent_traces | outcome | choice | 0.357 | 0.357 | ❌ |
| agent_traces | needs_human | noul | 0.429 | 0.714 | ❌ |
| customer_service | department | choice | 0.750 | 0.400 | ✅ |
| customer_service | refund_requested | noul | 0.850 | 0.700 | ✅ |
| customer_service | frustration | score | 0.400 | 0.500 | ❌ |
| security_incidents | severity | choice | 0.562 | 0.312 | ✅ |
| security_incidents | actively_exploited | noul | 0.438 | 0.688 | ❌ |
typed-decisionsrank 9 of 9
Full leaderboard →| Configuration | Mode | Decisions | Beats baseline | ECE |
|---|---|---|---|---|
| default payloadheadline | systemone | 0.343 | 2/20 | 0.196 |
| Noul phrased as a question (ablation)ablation | systemone | 0.336 | 3/20 | 0.218 |
Per-question accuracy — default payload, systemone mode, 2,000 judgments
| Task | Question | Type | Accuracy | Baseline | Beats? |
|---|---|---|---|---|---|
| td_agent_trace_observability | action | choice | 0.310 | 0.360 | ❌ |
| td_agent_trace_observability | needs_review | noul | 0.580 | 0.550 | ✅ |
| td_agent_trace_observability | outcome | choice | 0.400 | 0.500 | ❌ |
| td_agent_trace_observability | risk | score | 0.340 | 0.340 | ❌ |
| td_agent_trace_observability | urgency | score | 0.370 | 0.370 | ❌ |
| td_customer_service | action | choice | 0.050 | 0.520 | ❌ |
| td_customer_service | category | choice | 0.480 | 0.380 | ✅ |
| td_customer_service | churn_risk | score | 0.380 | 0.470 | ❌ |
| td_customer_service | needs_human | noul | 0.670 | 0.670 | ❌ |
| td_customer_service | urgency | score | 0.300 | 0.420 | ❌ |
| td_invoice_processing | discrepancy_severity | score | 0.010 | 0.470 | ❌ |
| td_invoice_processing | disposition | choice | 0.440 | 0.450 | ❌ |
| td_invoice_processing | duplicate | noul | 0.110 | 0.890 | ❌ |
| td_invoice_processing | matches_order | noul | 0.490 | 0.540 | ❌ |
| td_invoice_processing | urgency | score | 0.320 | 0.510 | ❌ |
| td_security_incidents | credential_compromise | noul | 0.550 | 0.550 | ❌ |
| td_security_incidents | disposition | choice | 0.050 | 0.740 | ❌ |
| td_security_incidents | severity | score | 0.310 | 0.490 | ❌ |
| td_security_incidents | true_positive | noul | 0.520 | 0.730 | ❌ |
| td_security_incidents | urgency | score | 0.170 | 0.500 | ❌ |
Fitted temperature per primitive (fold 0, judged on fold 1): choice 4, score 5, noul 7 — ECE 0.194 → 0.020, accuracy → 0.343.
Published numbers
Reported by the authors or by third parties, on their own suites. Not comparable to the leaderboard above; attributed so you can check them.
Third partynavinpai
Decoding Jev
Measured on: 400 intent, 400 MMLU-Pro, 100 relational questions
- Intent / MMLU-Pro / relational
- 63.0% / 13.5% / 8.0%
Third-party; method described, code not confirmed.
Model cardConvai Innovations
LAYA model card
Measured on: LAYA's own in-task and held-out families
- In-task macro accuracy / ECE
- 0.838 / 0.060
- Zero-shot held-out families
- 0.651 / ECE 0.207
- Intent / moderation / emotion
- 0.991 / 0.967 / 0.906
- Latency
- 38.4 ms p50 on a T4
Author-reported. The card's 83.8% vs Jev 67.8% compares two different benchmarks.
Model cardConvai Innovations
LAYA typed-decisions model card
Measured on: typed-decisions test split (400 cases, 2,000 decisions)
- Accuracy (no fine-tune)
- 0.362
Author-reported; the Jev row is quoted as "published", not measured by them. Trained on the suite's own train split.
Model cardConvai Innovations
LAYA BENCHMARKS.md
Measured on: AG News, DAIR Emotion, Banking77
- AG News / Emotion / Banking77
- 0.953 / 0.600 / 0.492
- ECE after temperature
- 0.081 (Jev quoted 0.246)
Reproducible for LAYA; the Jev rows are second-hand (nibzard and others) and used 72 Banking77 labels vs 77.
Frequently asked questions
How does LAYA compare to Jev?
On typed-decisions LAYA scores 0.343 against Jev's 0.726, beating the majority baseline on 2 of 20 questions (Jev: 18/20).
Is LAYA open source?
Yes. convaiinnovations/laya is released under Apache 2.0 and runs on your own hardware with the laya library.
Can LAYA extract entities?
No. LAYA answers questions about text and never returns spans; use a GLiNER-family model for extraction.