Security incident triage
Alerts and incident write-ups: severity, active exploitation, credential compromise, true positive, disposition, urgency.
On Metatext decisions, Jev leads with 0.906 mean accuracy against a 0.500 majority baseline.
Metatext decisions
Every model is zero-shot on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.906 | 0.500 | 2/2 |
| 2 | GLiClass large v3.0Knowledgator | 0.688 | 0.500 | 2/2 |
| 3 | GLiNER2 largeFastino | 0.625 | 0.500 | 2/2 |
| 4 | GLiNER 2.5 baseFastino | 0.625 | 0.500 | 2/2 |
| 5 | GLiClass modern large v3.0Knowledgator | 0.625 | 0.500 | 2/2 |
| 6 | GLiClass multilang ultraKnowledgator | 0.594 | 0.500 | 1/2 |
| 7 | GLiClass base v3.0Knowledgator | 0.594 | 0.500 | 1/2 |
| 8 | GLiGuard 300MFastino | 0.594 | 0.500 | 2/2 |
| 9 | GLiNER 2.5 multiFastino | 0.563 | 0.500 | 2/2 |
| 10 | GLiClass instruct large v1.0Knowledgator | 0.531 | 0.500 | 1/2 |
| 11 | GLiNER2 baseFastino | 0.500 | 0.500 | 0/2 |
| 12 | LAYAConvai Innovations | 0.500 | 0.500 | 1/2 |
| 13 | Vonwfzyx | 0.500 | 0.500 | 1/2 |
| 14 | GLiNER 2.5 smallFastino | 0.469 | 0.500 | 1/2 |
| 15 | GLiClass instruct base v1.0Knowledgator | 0.469 | 0.500 | 0/2 |
| 16 | LAYA typed-decisionsConvai Innovations | 0.438 | 0.500 | 1/2 |
| 17 | LAYA multilingualConvai Innovations | 0.438 | 0.500 | 1/2 |
| 18 | OpenJEVopen-jev | 0.406 | 0.500 | 1/2 |
| 19 | GLiNER multitask v1.0Knowledgator | 0.406 | 0.500 | 0/2 |
| 20 | GLiNER2 multiFastino | 0.375 | 0.500 | 0/2 |
| 21 | GLiClass edge v3.0Knowledgator | 0.344 | 0.500 | 0/2 |
typed-decisions
Every model is zero-shot except LAYA typed-decisions, which was fine-tuned on this suite.
| # | Model | Mean accuracy | Baseline | Beats baseline |
|---|---|---|---|---|
| 1 | JevTypeSafe | 0.756 | 0.602 | 5/5 |
| 2 | LAYA typed-decisionsConvai Innovations | 0.728 | 0.602 | 4/5 |
| 3 | GLiNER2 largeFastino | 0.620 | 0.602 | 3/5 |
| 4 | Vonwfzyx | 0.464 | 0.602 | 0/5 |
| 5 | GLiClass large v3.0Knowledgator | 0.462 | 0.602 | 1/5 |
| 6 | LAYA multilingualConvai Innovations | 0.404 | 0.602 | 1/5 |
| 7 | GLiNER 2.5 baseFastino | 0.398 | 0.602 | 0/5 |
| 8 | GLiClass multilang ultraKnowledgator | 0.390 | 0.602 | 0/5 |
| 9 | LAYAConvai Innovations | 0.320 | 0.602 | 0/5 |
The questions
- security_incidents.severity · choice · our tasksWhat is the severity of this security incident?critical severity · high severity · medium severity · low severity
- security_incidents.actively_exploited · noul · our tasksThere is evidence this issue is being actively exploited right now
- td_security_incidents.credential_compromise · noul · LAYA's public suiteThe evidence indicates a credential or account has been compromised.
- td_security_incidents.disposition · choice · LAYA's public suiteHow should this security alert be dispositioned?close_benign · contain · investigate · monitor
- td_security_incidents.severity · score · LAYA's public suiteHow severe is the potential impact if this alert is real?Negligible: no access to anything sensitive. · Low: limited access, easily reversed. · Moderate: access to internal systems or non-public data. · High: access to production, secrets or customer data. · Critical: active compromise of crown-jewel systems.
- td_security_incidents.true_positive · noul · LAYA's public suiteThis alert reflects genuinely malicious or unauthorised activity.false · true
- td_security_incidents.urgency · score · LAYA's public suiteHow quickly must a responder act on this alert?No time pressure; can wait indefinitely. · Routine; handle within the normal queue. · Elevated; should be handled within the same week. · Critical; requires action within the same day.
Frequently asked questions
What does security incident triage involve?
Alerts and incident write-ups: severity, active exploitation, credential compromise, true positive, disposition, urgency.
Which model is best at security incident triage?
Jev leads on Metatext decisions at 0.906 mean accuracy over 2 questions.
Related
Support tickets: which team, is a refund requested, how frustrated, churn risk, urgency, does a human need to step in. The domain TypeSafe uses in its own docs.
Traces of an autonomous agent run: outcome, risk, urgency, what the observability system should do, whether a human should review it.
Invoices against purchase orders: does it match, is it a duplicate, discrepancy severity, disposition, urgency.