Published
A 100-case security triage demo put Jev at 65.3% balanced accuracy against Terra 75.1% and Opus 91.2%
RZ posted a recorded comparison of Jev, OpenAI Terra in Codex, and Claude Opus in Claude Code on 100 synthetic TypeScript security cases, five passes each. Jev's balanced accuracy was 65.3% at 0.57 s median. Terra 75.1% at 10.23 s. Opus 91.2% at 13.83 s. Jev repeated the same answer on 95 of 100 cases. Cost about $0.015 versus $15.02 and $13.94 for 500 decisions.
RZ posted a cybersecurity experiment on September 21. The repo is Robertzu43/system-one-security-triage, MIT. A static dashboard at system-one-security-triage.rzuniga-9b4.workers.dev serves the saved run. The page does not call a model.
The published numbers come from results/recorded/2026-09-19-public-v5. One hundred synthetic TypeScript and Node.js cases, five passes each, 1,500 decisions, zero errors. Generated September 19, 2026 at 20:47 UTC. Jev 1.13.0 ran through @typesafe-ai/sdk 0.6.0. OpenAI gpt-5.6-terra ran through Codex CLI 0.147.0. Anthropic claude-opus-4-6 ran through Claude Code 2.1.278. Earlier v2, v3, and v4 directories were recorded on a corpus that leaked labels through file paths or left broken-access-control cases undecidable. The builder rejects those by corpus hash.
The frozen set covers injection (34), broken access control (33), and SSRF (33). Expected outcomes are balanced: 34 vulnerable, 33 safe, 33 insufficient context. Span paths carry no disposition words. Private context-resolution metadata is never sent. Jev answers one five-way typed Choice. Terra and Opus return structured final answers built from the same outcome definitions. The selected label is scored the same way for all three. No confidence threshold sits between Jev’s choice and the score.
Balanced accuracy, the unweighted mean of per-class recall, is 65.3% for Jev, 75.1% for Terra, and 91.2% for Opus. Model-request medians: Jev 0.57 seconds, Terra 10.23 seconds, Opus 13.83 seconds. The same answer across all five passes: Jev 95 of 100 cases, Terra 76, Opus 91. Jev’s class recall was 93.9% on insufficient context, 45.5% on safe, and 56.5% on vulnerable. False-safe rate was 0 for every evaluator. Family accuracy on the vulnerable cases Jev labelled was 1.00; when it called a span vulnerable it named the right family.
Cost for 500 decisions: Jev $0.015146 on TypeSafe’s published rate card (360,620 input tokens), Terra $15.017 on a rate card, Opus $13.94 as the provider billed. Jev’s multiclass Brier score is 0.472. Frontier-model confidence is not compared with that.
The README calls the page a descriptive synthetic demo. Expected labels were written by one author. At 100 cases a gap under about fifteen points is not distinguishable from noise, and per-class figures rest on about thirty cases each. The corpus is balanced by construction; real code is mostly not vulnerable, so precision here does not transfer. Latency compares a Jev HTTP call with frontier models driven through agent CLIs.
We did not rerun the record script.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 21 thread from @snoopydev99 is the primary post. The repo is Robertzu43/system-one-security-triage, MIT. The live dashboard serves results/recorded/2026-09-19-public-v5: 100 synthetic TypeScript/Node.js cases, five passes, 1,500 decisions, zero errors, generated 2026-09-19 20:47 UTC. Evaluators: Jev 1.13.0 through @typesafe-ai/sdk 0.6.0; OpenAI gpt-5.6-terra through Codex CLI 0.147.0; Anthropic claude-opus-4-6 through Claude Code 2.1.278. Families: injection 34, broken access control 33, SSRF 33. Outcomes: vulnerable 34, safe 33, insufficient context 33. Evidence is label-neutral. Jev answers one five-way Choice; the selected option is scored the same way as Terra's and Opus's structured answers, with no confidence threshold. Balanced accuracy (unweighted mean of per-class recall): Jev 65.3%, Terra 75.1%, Opus 91.2%. Model-request p50: Jev 570 ms, Terra 10.23 s, Opus 13.83 s. Same answer on all five passes: Jev 95/100, Terra 76/100, Opus 91/100. Jev class recall: insufficient context 93.9%, safe 45.5%, vulnerable 56.5%. False-safe rate 0 for all three. Cost for 500 decisions: Jev $0.015146 on TypeSafe's rate card, Terra $15.017 on a rate card, Opus $13.94 as billed. Earlier v2, v3, and v4 runs were superseded because labels leaked through paths or access-control cases were undecidable. The README calls this a descriptive synthetic demo. At 100 cases a gap under about fifteen points is not distinguishable from noise. Latency compares a Jev API call with frontier models driven through CLIs. We did not rerun it. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; the figures are reported as published.
Compare
LangChain's judge eval put Jev next to Terra, Luna, and Sonnet on five frozen traces; Jev matched 500/500 binary labels there. SREGym-Lite used Jev to gate a coding agent. This set is security dispositions on short TypeScript spans. Balanced accuracy sat below Terra and Opus. Median request time, billed cost, and five-pass agreement sat above them. Safe-class recall is the weak cell (45.5%). Lemkin's 32% false admissions were a different miss on a different job.
Terms
- Balanced accuracy
- Unweighted mean of per-class recall over vulnerable, safe, and insufficient context. The published v5 run puts Jev at 65.3%, Terra at 75.1%, and Opus at 91.2%.
- 2026-09-19-public-v5
- The frozen recorded run on RZ's dashboard. One hundred synthetic cases, five passes, 1,500 decisions. Earlier v2 through v4 directories are rejected by corpus hash.