Updated
Desk
Evals
Measurements this digest has already published, with the caveat that came with each one. Vendor tables sit next to independent write-ups and personal clips. We did not rerun any of them. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; the numbers below are what people posted. Side-by-side tables are on compare. The Jev benchmark page separates JevBench from TypeSafe's own workflow test. Decision model hallucination keeps the RAGTruth rows on their own tests. Stories grouped by tag are on topics.
OR-decision-4B, JevBench 193 of 231
83.55% · Choice 79.7 · sarcasm 53.6 · Apache-2.0 LoRA
alphaXiv, October 30, 2026, 16:06 UTC. Hugging Face card JevBench primary window 193 of 231, 83.55%. Launch chart Choice 79.7 against Jev 79.6, Score 67.0 against 65.4, Yes/No 88.3 against 87.4, sarcasm 53.6 against Jev 70.7. Apache-2.0 LoRA on Qwen3.5-4B, 117,186 train examples. Do not merge 83.55 with 79.7 or with Decision Index. This desk did not run the weights.
MiniCPM5-2B-Jev, JevBench 78.79%
78.79% · first open ≤2B · decision-v7 86.41% · ECE 0.0307
OpenBMB, October 28, 2026, 13:01 UTC. Project prints 78.79% on JevBench, first among open models at or under 2B. The same README table prints hosted Jev 1.13.0 at 86.6%. decision-v7 prints 86.41% accuracy and ECE 0.0307. Apache-2.0 on MiniCPM5-2B. Weights ytbai/MiniCPM5-2B-Jev. Do not merge 78.79 with 86.41. This desk did not run the weights.
BranchOne, A100 Colab median 0.305 s
0.305 s · Qwen3.5-0.8B-Base · Choice probs · no JevBench
Necati Demir, October 29, 2026, 13:31 UTC. Open checkpoint returns one probability per supplied candidate from Qwen3.5-0.8B-Base plus LoRA. Author A100 Colab median 0.305 s on ten short text questions. Apache-2.0. Weights ndemirdemir/BranchOne. No published JevBench score. 0.305 s is not Celeris 67 ms and not hunch 20/100 ms. This desk did not run the weights.
hunch, EmbeddingGemma2 System One on CPU
20 ms text · 100 ms images · Clef teacher · no public repo
Matt Mastracci, October 27, 2026, 15:09 UTC. Published clocks 20 ms on text and 100 ms on images for a CPU /v1/systemone endpoint on EmbeddingGemma2. A later thread says Cloudflare Clef teaches a local student in real time, and open-source launch was planned for later that week. Opened October 28, public search found no hunch GitHub repository and no hosted POST. This desk did not call an endpoint. 20 ms is not Clef-flash's October 1 median of 38.8 ms.
d3-ultra, Decision Index 0.4.1 Full 70.18
70.18 rank 2 · SR-Text 87.2 · vision 70.59 · HTTP 401
Xunzhuo Liu, October 27, 2026, 03:01 UTC. Opened October 26, Decision Index 0.4.1 Full 70.18, second of 229, +3.03 over d3 at 67.15. The card's private SR-Text mean is 87.2. Vision board edition 0.4 full 70.59, rank 11 of 56. Date headers 06:25:57 GMT and 06:25:58 GMT October 27, GET /v1/models listed d3-ultra, POST without a key returned HTTP 401. Apache-2.0. This desk did not run the weights.
d3-turbo, Decision Index 0.4.1 Full 67.31
67.31 · vision 72.46 rank 2 · 2.12 s / 255 · HTTP 401
Xunzhuo Liu, October 20, 2026, 00:42 UTC. Opened October 26, Decision Index 0.4.1 Full 67.31, eleventh of 229. Vision board edition 0.4 full 72.46, second of 56. The card prints 255 questions about one photo at 2.12 s against d3-flash at 32.94 s. The free API aliases vllm-sr/d3 to d3-turbo. Date header 06:25:57 GMT October 27, GET /v1/models listed that alias. This desk did not run the weights.
Decision Model Factory, DF-JA-Large 67.63
67.63 · public high 60.05 · runner 2.44× · no public weights
Yuma Ichikawa, October 27, 2026, 03:19 UTC. arXiv 2610.34956. Table 1 DF-JA-Large S1MB Japanese Task Avg 67.63 against a fixed public snapshot high of 60.05. DF-Long shared-state runner 29.4 ms against 71.7 ms, 2.44 times. A Hugging Face search on October 27 returned no matching public checkpoint. This desk did not run a build.
GLiDE v1.2, Decision Index 0.4.1 Full 71.5
71.5 rank 1 · $0.04 · JevBench 76.4 rank 3 · HTTP 401
Fastino, October 22, 2026, 16:36 UTC. Opened October 26, Decision Index 0.4.1 Full 71.5, rank 1 of 229, same-skill 68.3, new-domain 76.3. Docs $0.04 per million input tokens. JevBench Fastino GLiDE (hosted API, 22 Oct 2026) composite 76.4, rank 3 of 48. An older GLiDE row stays 30.6 at $0.137. Date headers 08:49:09 GMT to 08:49:11 GMT, GET /v1/models omitted GLiDE, POST /v1/systemone HTTP 401. 64.81 stays the October 1 0.2.1 self-run.
jev-latest named jev-1.14.0
jev-1.14.0 · both aliases · named 1.13.0 still 1.13.0
Date headers 08:49:11 GMT to 08:49:40 GMT October 26. GET /v1/models listed jev-latest and jev-preview. POSTs with those aliases named jev-1.14.0. A named jev-1.13.0 POST named jev-1.13.0. Docs say both aliases point at jev-1.14.0, and that there is no preview build. At 00:28 UTC October 16, jev-latest still named jev-1.13.0.
Abe, 1,774 posts, Jev $0.055 against AI_FILTER $1.87
1/34 · AUC 0.9685 · 54 of 70 · desk noul 0.04
ABE Masanori, October 24, 2026, 11:20 UTC. 1,774 of his 2025 X posts, run 2026-10-20. Jev Noul about $0.055 against AI_FILTER about $1.87. AUC 0.9685. Of 70 splits where Jev scored under 0.2 and AI_FILTER was TRUE, he judged Jev right on 54. Response jev-1.13.0. Date headers 08:49:12 GMT to 08:49:40 GMT October 26, this desk's jev-1.14.0 noul on his Lightdash example was 0.04. 0.04 is not his 0.03.
Jeva-4B, 0 flips on 600 Choice items
0 flips · BANKING77 91.62 · Jev 82.50 · Apache-2.0
DailyPapers, October 14, 2026, 20:10 UTC. Qi Li and Xinchao Wang, arXiv 2610.18773. Table 1 BANKING77 K=16 91.62 against Jev-1.13.0 at 82.50. Typed Test 78.30. 600 Choice items, 0 flips, Jev-1.13.0 17.00%. Null AUROC 98.66%. Apache-2.0, Qwen3.5-4B. This desk did not run the weights.
mammoth-1, Decision Index 0.4 Full 71.24
71.24 · live 0.4.1 68.4 rank 3 · elephant-1 62.65 · HTTP 401
Base Labs, October 21, 2026, 17:14 UTC. Card Decision Index 0.4 Full 71.24, first of 208. Opened October 23, live space v0.4.1, 229 models, mammoth-1 Full 68.4 rank 3, public 66.0. elephant-1 card 62.65, live Full 60.3. MIT, Qwen3.5-27B and Qwen3.5-9B. Date headers 02:23:21 GMT to 02:23:23 GMT, GET /v1/models with no key returned HTTP 401. 71.24 is not 68.4.
OpenAI Decisions API, generally available
HTTP 200 · gpt-6-luna · 1.0 / 0.99 · 400 questions · $0.10 / $0.01
OpenAI Developers, October 22, 2026, 19:07 UTC. Date headers 02:23:12 GMT and 02:23:13 GMT October 23, two POSTs returned HTTP 200. Urgency predicate 1.0, 182 input tokens, openai-processing-ms 899, desk 1,584 ms. Three questions chose technical at 0.99, 444 input tokens, processing 165, desk 977 ms. Guide uncached $0.10, cached $0.01, output free, up to 400 questions.
SubQ Snap 1.1, Artificial Analysis Decision Index 65
65 tied · $0.04 · noul 0.999 · subq-snap-1.1
Alexander Whedon, October 22, 2026, 19:48 UTC. Artificial Analysis prints 65, tied with d1-ultra and Elephant 9B, Jev 1.14 at 57. Docs $0.04 per million input tokens. Date headers 02:23:17 GMT to 02:23:20 GMT October 23, three POSTs HTTP 200. GET /v1/models listed subq-snap-1.1 and subq-snap-preview. 65 is not the October 12 61.7 cell.
d1-ultra, ClaimBench 1,960 of 1,960
1,960 of 1,960 · 99.9% alone · AA 65 stays
Vals AI, October 22, 2026, 23:57 UTC. d1-ultra is the only decision model to get all 1,960 ClaimBench claims right. The page, opened October 23, prints 99.9% alone at a 1% error budget, 0.00% wrong. October 20 ranked Decider V1.2 first at 99.69%. 1,960 is not Artificial Analysis 65.
Drex Max, Decision Index 0.3.2 public 67.59
67.59 · Jev 1.14 60.70 · $0.06 · HTTP 401
Nace, October 19, 2026, 13:33 UTC. Product page, opened October 20: Decision Index 0.3.2 public 67.59 against Jev 1.14 at 60.70, 37 benchmarks, 140,178 scored requests. Docs $0.06 per million input tokens, model drex-max. Date headers 08:50:01 GMT and 08:50:02 GMT, GET /v1/models with no key returned HTTP 401. 34 ms is the post. 183 ms is the product table's median. 67.59 is not Drex 1.5's 58.08.
StartLux-Decision v1.1 27B Instant, Full score 69.12
69.12 · first of 174 · public 72.88 · 0.2.1 stays 63.88
README release notes dated 2026-10-16, opened October 20. Decision Index 0.3.2 Full score 69.12, first of 174, public 72.88, same-skill private 64.82, new-domain 61.41. StartLux posted 69.1 on October 17, 04:34 UTC. 63.88 stays the October 3 self-run of edition 0.2.1. 72.83 is the models table Instant cell. We did not rerun the board.
Datadog counterfeit policy, Jev Noul recall 3.88%
82.90% to 3.88% · is_risky 0.893 to 0.290 · desk 0.94 then 0.13
Datadog Security Labs, dated October 15, posted October 19, 19:12 UTC. Four datasets, 4,643 attacks. Noul has_prompt_injection recall 82.90% raw, 3.88% with a counterfeit policy prepended. LangChain is_risky 40-run mean 0.893 then 0.290. Date headers 08:50:46 GMT and 08:50:47 GMT October 20, this desk's jev-1.13.0 is_risky was 0.94 then 0.13 on a shortened handbook. A dump-the-prompt Noul stayed 0.99 then 0.92. 0.13 is not 0.290. 0.92 is not 3.88%.
Sieve-31B, Decision Index 0.3.2 Full score 63.8
63.8 · public 62.52 · Cuth second · lab tied 3rd of 174
Hugging Face card, opened October 20, 2026. Full score 63.8 on edition 0.3.2, board data of October 14. Authors' public-suite run 62.52. Apache-2.0 LoRA on Gemma 4 31B. Cuth, 01:50 UTC, prints second globally. sthanika.ai prints 63.81, tied 3rd of 174. This desk did not run the weights. 63.8 is not StartLux 69.12 and not Drex Max public 67.59.
Clairic cascade, 19.3 s against 61.7 s
19.3 s / 61.7 s · $6,032 / $11,740 · $0.10
Michelle Chen, October 19, 2026, 17:54 UTC. Cloudflare blog table: Clairic then Clef then GLM-5.3 against GLM-5.3 at every step, 19.3 s against 61.7 s, $6,032 against $11,740 per million claims. Extract 0.46 s, decide 0.22 s. Docs $0.10 per million input tokens, @cf/cloudflare/clairic. Michelle's follow-up prints 3.5x and 77% cheaper for a live demo. Those two prints stay separate. This desk did not call the API.
basal-omni-1.0-ultra, JevBench Capability 82.5
82.5 · open cap. rank 1 of 184 · composite 71.8 rank 3 · Werdykt 0.879
Opened October 19, 2026, about 07:20 UTC. JevBench JSON revision v1.6.29, 184 ranked open-weight systems. basal-omni-1.0-ultra Capability 82.54, Intelligence 74.06, Calibration 91.02, first of 184 on that axis. Composite 71.76, open composite rank 3. H2O-Lightning-4B v1.1 is open composite rank 1 at 72.52. Remek Kinas posted October 17, 14:15 UTC. The card prints Werdykt v1 at 0.879. Apache 2.0, Gemma 4 31B. This desk did not run the weights. 82.5 is not Werdykt 0.879.
HiddenLayer checkout ad, Jev 91% Confirm Price
91% / 9% · desk 1.0 then 0.89 · 30 toxic prompts
HiddenLayer, October 14, 2026. TypeSafe Jev API on jev-ultrafast: Confirm Price 91 percent, Place order 9 percent after a checkout ad. The in-house Qwen analog moved Place order from 99.7 percent to 19.3 percent. Date headers 07:24:57 GMT and 07:24:58 GMT October 19, this desk's jev-1.13.0 still chose place_order, 1.0 then 0.89 with confirm_price at 0.11. Those two POSTs are not the 91 / 9. Decision hijacking 1.8% to 3.5% is a different paper.
Qwen-Audio-Jev, MMAU-mini 78.70%
78.70% · MMSU 84.06% · EER 6.50% · 20x
arXiv 2610.21614, submitted October 15, 2026. 1LittleCoder posted October 18, 21:14 UTC. Choice 78.70% on MMAU-mini and 84.06% on MMSU. Noul EER 6.50% on VoxCeleb1-O. Score NISQA nMAE 0.1062. About 20x against Qwen-Audio-3.1-ASR-Flash-Next. This desk did not run the model. ToddlerJev reprints those three names as published cells.
ToddlerJev, MMAU-mini 68.70%
68.70% · MMSU 61.30% · MMAR 56.90% · 111.9 ms
Vyom, October 18, 2026, 22:19 UTC. Hugging Face TeenageWorks/Toddler-Jev, Qwen Research License. MMAU-mini 68.70 on 1,000 items. Streaming p50 111.9 ms on an A100 40 GB. Silence diagnostic 38.50% against 73.50% with full audio. This desk did not run the weights. 68.70 is not Qwen-Audio-Jev's 78.70.
Jev 1.14.0, JevBench API composite 70.2
70.2 · rank 9 of 38 · Capability 75.01 · jev-latest still 1.13.0
Opened October 19, 2026, about 07:20 UTC. JevBench JSON revision v1.6.29, 38 ranked API offerings. Jev 1.14.0 composite 70.20, Capability 75.01, Intelligence 59.94, Calibration 90.08, p50 0.224 s. Jev 1.13.0 stays seventh at 71.49, Capability 77.13. Date header 07:24:57 GMT, this desk's jev-latest still named jev-1.13.0. 70.2 is not TypeSafe's internal 60.0.
kettei-fat, JevBench API composite 77.8
77.8 · rank 1 of 39 · capability 86.3 · Jev 1.13.0 71.5 rank 7
Opened October 17, 2026, about 01:48 UTC. JevBench API board release v1.6.1, revision v1.7.46, updated 2026-10-16, 39 ranked API offerings. kettei-fat composite 77.8, capability 86.3, estimated $0.0260 per 1,000, median 0.20 s, Intelligence 79.8. v1.6.27 draw measured 2026-10-16, put on the v1.6.1 scale. Federica's post printed #1 of 37 and intelligence 80 against 70. ImageJevBench v0.3.0 capability 75.9 and composite 75.4 are a different board. The blog's 93.1% on 231 public items is a third print. GET /v1/models with no key returned HTTP 401. This desk did not send a billed POST.
pplx-decider-v1.2, Decision Index 0.3 overall 68.09
68.09 joint · v1.1 62.30 · flash 60.05 · $0.02 / $0.01
Perplexity, October 16, 2026, 18:42 UTC. Card table, public suite. Joint requests 68.09 against v1.1-27b at 62.30. One question per request 67.75. Vision 77.43 against 73.22. Flash one-question overall 60.05, joint 58.88, vision 72.53. Docs price v1.2 at $0.02 per million input tokens and flash at $0.01. The October 8 space print of v1.1 at 62.8 stays that reading. 85.71% on the 7,210-row panel is the October 1 chart. GET of the OpenRouter ids returned HTTP 404. This desk did not call Perplexity.
Vercel Gateway, Jev two-week reach 35.3%
35.3% · Sonnet 5.5 18.1% · spend 0.27% / 0.3% / 0.15%
Vercel AI Gateway Production Index, October 2026, published October 15. Launch table: TypeSafe AI Jev, gateway launch Sep 17, reach 35.3% of paid teams in the first two weeks, against Claude Sonnet 5.5 at 18.1%. Summary spend 0.27% in that window. Scorecard spend share 0.3%. A later paragraph prints 1% of September tokens and 0.15% of spend. TechCrunch, October 16, 15:33 UTC, rounds reach to 35%. The September 18 post still prints nearly 13% at 24 hours. We did not pull the underlying table.
Celeris-1 Decision, 80.0% on 3,210 jev-bench items
80.0% · Jev 1.13.0 73.1% · 67 ms · AA 52
Celeris blog, October 8, 2026. Praveenrajus/jev-bench, 22 datasets, 3,210 items, harness 0cfd276. celeris-1-decision 0.800 against Jev 1.13.0 at 0.731, pplx-decider-v1.1-27b 0.790, Mercury Decide 0.753, gpt-6-luna 0.690. Median from AWS us-east-1 is 67 ms for one question. Input $0.04 per million tokens, output free. Opened October 16, Artificial Analysis Decision Index lists it at 52. Date headers 15:19:34 GMT to 15:19:36 GMT, GET /v1/models with no key returned HTTP 401. This desk did not send a billed POST. 80.0% is not Florian's JevBench composite.
Artificial Analysis Decision Index, d1-ultra 65
65 · Jev 1.14 57 · hosted d1 50 · Celeris 52
Opened October 16, 2026 UTC. Artificial Analysis Decision Index, 11 of 13 models, language models left off. d1-ultra 65, Jev 1.14 57, GLiDE 54, Microsoft-Decision-1 54, Decider v1.1 27B 53, Mercury Decide 53, Celeris-1 Decision 52, hosted d1 50, Clef 49, GPT-6 Luna (Decisions) 48, Clef-Omni 44. Liquid posted d1-ultra October 15, 17:00 UTC, and priced input at $0.042 per million tokens. This desk did not call the API. 65 is not Hugging Face Decision Index 0.2.1 or 0.3, and it is not Liquid's 112-evaluation average of 76.7.
AA-Omniscience-Decision, d1-ultra non-hallucination 85%
85% · d1 2% · accuracy 32% against 41% · 92 ms
Artificial Analysis, October 16, 2026, 11:11 UTC. AA-Omniscience-Decision non-hallucination 85% against 2% for hosted d1, accuracy 32% against 41%. Microsoft-Decision-1 is second on that non-hallucination row at 40%. Time per Task 92 ms on the first tweet, 93 ms on the follow-up, behind Jev 1.14 at 78 ms and Celeris-1 Decision at 63 ms. Cost per 1M tasks $32. This slice is not the composite 65.
d3 Decision Index 0.3.2, 66.3
66.3 · October 11 64.7 · public 65.5
Opened October 16, 2026 UTC. The d3 card prints Decision Index 0.3.2 at 66.3, public 65.5, same-skill 62.8, new-domain 60.5, first on the official text board. The October 11 reading of that card was 64.7 on edition 0.3.1. Xunzhuo Liu posted a Box demo at 11:35 UTC. The card still lists Choice, Yes / No, and Score. This desk did not run the weights.
Liquid 112-evaluation average, 76.7 against Jev 1.14 at 69.4
76.7 · 69.4 · d3 73.5 · calibration error 1.5
Liquid AI, October 15, 2026, 17:00 UTC. Eight-category table on the d1-ultra thread. Average 76.7 against Jev 1.14 at 69.4, d3 at 73.5, pplx-decider v1.1 at 70.3, Clef at 67.0, Microsoft-Decision-1 at 66.6, GPT-6 Luna (Decisions) at 60.9. Pictures cells for Microsoft-Decision-1 and Luna are blank. Calibration error on that chart is d1-ultra 1.5, d3 2.6, Jev 1.14 2.7. This table is not the Artificial Analysis score of 65.
Span-02 Decision Index 0.3, thinking off 61.2
61.2 · Jev 58.0 · card text 63.62 · no limit 64.1
Respan, October 15, 2026, 17:00 UTC. Blog, opened October 16: local rebuild of Decision Index 0.3 public, 37 benchmarks. Thinking off 61.2 against Jev 1.13.0 at 58.0. 1 s 63.0, 3 s 63.6, no limit 64.1. Hugging Face card text index 63.62 on 140,178 requests, vision 70.69 on 12,210. Those two prints stay separate. This desk did not run the weights. 61.2 is not d3's 64.7 and not Artificial Analysis 65.
OpenRouter Armor, 446 of 492
90.7% · 446 of 492 · $0.0021 / call · this desk noul 0.98 and 0.01
OpenRouter, October 15, 2026, 21:00 UTC. Internal 492 tool calls, synthetic set seeded from harness logs. Jev 1.13 Armor default 90.7%, 446 of 492, $0.0021 a call. Clef 435 of 492. Sonnet 5 434 of 492. GPT-6 Luna decisions 407 of 492. This desk did not rerun the 492. Date headers 01:18:32 GMT and 01:18:33 GMT October 16, two POSTs to jev-1.13.0. git push origin main noul 0.98. ls ./reports noul 0.01. Those two calls are not the 492.
Every-vehicle Box, one rectangle, n = 3
one box · 0.19 / 0.17 / 0.13 · type array HTTP 400 · Elon frame six at 0.59
This desk, October 16, 2026, Date headers 08:19:44 GMT to 08:19:46 GMT. Three jev-1.14.0 Boxes, instructions every vehicle, on a 400 by 400 three-car PNG, each returned one rectangle, confidence 0.19, 0.17, 0.13. A POST with type array, items type box, returned HTTP 400, Invalid request. Named Boxes in one request scored IoU 0.89, 0.89, 0.90. Count chose three at 1.0. A 960 by 540 frame from Elon Musk's flying-cars clip chose six at 0.59, in-air noul 0.94, flying box confidence 0.69. TypeLLM's cookbook at 14.00 s prints six vehicle boxes, including Tesla's flying Roadster. This desk did not call TypeLLM. Five POSTs printed 5,187 input tokens, $0.000218.
Aviz Maeir Box, 82% on 180 scenes
82% · 0.21 s · $0.15 per 1k · ≥0.7 right 107 of 108
Aviz Maeir, October 15, 2026, 22:04 UTC. Same 180 scenes against OpenAI Responses API plus strict JSON. Jev 1.14 82 percent, 0.21 s, $0.15 per 1k. gpt-6-luna 84 percent, 1.5 s, $0.22 per 1k. gpt-6.1-sol 95 percent, 3.4 s, $5.03 per 1k. Confidence 0.7 or above was right 107 of 108. Luna printed 0.9 to 1.0 on 27 confident misses. This desk did not rerun the 180. Date headers 00:28:47 GMT to 00:28:48 GMT October 16, this desk's Continue button n = 3 averaged IoU 0.93, confidence 0.91. Four colors in one request used 4,458 input tokens against 6,612 as four separate POSTs. 82 percent is a different test from Continue IoU 0.93 and from TypeSafe's 30 of 36 crops.
Continue button Box, n = 3, mean IoU 0.93
0.93 IoU · conf 0.91 · noul 0.99 · HTTP 400 on 1.13.0
This desk, October 16, 2026, Date headers 00:28:47 GMT to 00:28:48 GMT. 400 by 400 Checkout PNG, green Continue button drawn at 0.55 to 0.90 and 0.70 to 0.88. Three jev-1.14.0 boxes averaged left 0.55, right 0.91, top 0.71, bottom 0.88, confidence 0.91. IoU 0.92, 0.92, 0.94. Noul 0.98, 0.99, 0.99. Matching jev-1.13.0 Noul 1.0. The same Box on jev-1.13.0 returned HTTP 400. An empty field scored noul 0.02 and box confidence 0.04. Aviz 82 percent is a different test.
Yamashita Kyoto-ben 60, 0.833 against 0.817
0.833 · 0.817 · majority vote same on 60 · this desk social_nicety n = 3
Yuuki Yamashita, October 15, 2026. Sixty frozen Kyoto-ben phrases, six labels, 420 calls, about $0.02. Three-run mean accuracy 0.817 on jev-1.13.0 and 0.833 on jev-1.14.0. Majority vote matched on all 60. McNemar p = 1.000. This desk did not rerun the 60. Date headers 14:57:06 GMT to 14:57:09 GMT, this desk's Choice on うちの子ぉも、ええ子にしてますえ, criteria written here, chose social_nicety on all three jev-1.14.0 calls, confidence 0.69, 0.66, 0.64, and on all three jev-1.13.0 calls, 0.33, 0.36, 0.34. n = 3. SosyalDijital 29 of 30 is a different test.
Jev4Mellea synthetic scenes, coverage 5 of 12
5 of 12 at 0.10 / 0.90 · accepted 0 of 6 · median 214.3 ms
SoundBlaster/Jev4Mellea, October 15, 2026, started 11:45:44 UTC. Twelve procedural scenes on jev-1.14.0, six expected accepts and six rejects, labels from geometry. At 0.10 / 0.90: FAR 0 of 6, FRR 0 of 6, uncertain 7 of 12, coverage 5 of 12, accepted 0. At 0.20 / 0.80 coverage 9 of 12. Adapter 0.6.0 from that checkout. This desk did not rerun the 12. Date headers 14:57:24 GMT to 14:57:27 GMT, this desk's two-square PNG scored blue noul 0.98 and red noul 0.56 on jev-1.14.0, n = 3. The 0.46 red-square mean is a different sample.
AI BENCHY Jev 1.13 vs Jev 1.14, 2.6 against 2.3
2.6 · 2.3 · 6 of 12 against 4 of 12
Cristian, October 15, 2026, 12:21 UTC. Opened later that UTC day, AI BENCHY prints Jev 1.13 default score 2.6, tests correct 6 of 12, attempt pass rate 26.1 percent, 698 ms average. Jev 1.14 default 2.3, 4 of 12, 23.2 percent, 377 ms. Both rows 36 of 69 attempts. Adapters supply choices. Each row supports 12 of 23 tests. Score uses the full suite. This desk did not rerun that suite. 2.3 is not JevBench 71.5 and not TypeSafe's 60.0 chart.
SosyalDijital Turkish tickets, 29 of 30
29 of 30 · same missed buyer · this desk seller n = 3 on 1.14.0
Göksu Güneş, October 15, 2026, 10:02 UTC. SosyalDijital sent 30 synthetic Turkish messages, labels buyer, seller, and other, one pass each, 60 API calls through OpenRouter. Both Jev 1.13 and Jev 1.14 scored 29 of 30. The shared miss is Reklam değil gerçek talep: ürün fotoğraflarını düzenleteceğim, ücret konuşalım. Expected buyer, both returned seller. Recorded model fields typesafe/jev-1.13-20260917 and typesafe/jev-1.14-20261014. Mean clocks 0.3669 s and 0.3518 s. Recorded cost $0.001129884. This desk did not rerun the 30. Date headers 11:04:57 GMT to 11:04:59 GMT, this desk's Choice on that sentence, written here, chose seller on all three jev-1.14.0 calls, confidence 0.48, 0.43, 0.52. Matching jev-1.13.0 calls chose seller, buyer at 0.28, then seller. n = 3. Chinese 1.30 is a different test.
Image tokens, 850 PNG
2,238 in state · 4,209 in two instructions · Wes Bos $0.000096936
Wes Bos, October 14, 2026, 23:15 UTC, billed $0.000096936 for one question on an 850 by 850 image, about 10,000 images a dollar. He wrote that two questions duplicate token usage, and that 1568 is 4x 784. Date headers 11:04:54 GMT to 11:04:57 GMT, this desk, jev-1.14.0, synthetic PNGs. One Noul, image in state: 784 used 1,884 input tokens, 850 used 2,238, 1568 used 6,589. Two Nouls on the 850 file in state used 2,259. Two Nouls that each named that file in instructions used 4,209. A matching 850 Noul on jev-1.13.0 used 2,235. n = 1 per layout. 6,589 / 1,884 is 3.50. The 0.46 red-square mean is a different sample.
jev-1.14.0, Chinese payouts n = 3
mean 1.30 at 0.54 · 1.13.0 mean 1.54 at 0.31
This desk, October 15, 2026, Date headers 06:16:21 GMT to 06:16:25 GMT. Same Chinese payouts sentence, English questions, three POSTs each. jev-1.14.0 frustration 1.27 at 0.59, 1.32 at 0.52, 1.32 at 0.52. jev-1.13.0 1.55 at 0.32, 1.52 at 0.28, 1.55 at 0.32. Language 1.0 and billing on all six. The 03:03 UTC singles 1.37 at 0.44 and 1.54 at 0.31 stay a separate sample. Thirty-two POSTs in that sweep, 20,518 input tokens, $0.000862 at list price.
Danger-zone image, red square n = 3
0.46 mean on 1.14.0 · 0.94 on 1.13.0
This desk, October 15, 2026, Date headers 06:16:36 GMT to 06:16:42 GMT. Same 400 by 400 PNG, red square from 0.40 to 0.80 of the width and 0.20 to 0.70 of the height. jev-1.14.0 Noul 0.49, 0.47, 0.42. jev-1.13.0 0.94, 0.94, 0.93. An empty field scored 0.04 and 0.01. A request with the present image and no TypeSafe-Danger-Zone-Images header returned HTTP 400. Box mean left 0.40, right 0.79, top 0.16, bottom 0.71, confidence 0.91. The 01:22 UTC singles 0.39 and 0.90 stay a separate sample.
jev-1.14.0, four-language payouts tickets
language 1.0 · billing on four · Chinese 1.37 at 0.44
This desk, October 15, 2026, Date headers 03:03:53 GMT to 03:04:52 GMT. Six POSTs with model jev-preview named jev-1.14.0, HTTP 200. Four payouts tickets in English, Chinese, Spanish, and Japanese, questions in English. Language Choice 1.0 on the matching label for each. Department billing 0.86, 0.99, 0.96, 0.99. Urgency 0.95, 0.96, 0.93, 0.94. Frustration 1.02 at 0.97, 1.37 at 0.44, 1.02 at 0.97, 1.10 at 0.85. A Chinese API-500 outage chose technical at 1.0. Chinese questions on the Chinese payouts sentence stayed billing at 0.99, frustration 1.33 at 0.50. Five matching jev-latest POSTs named jev-1.13.0 and also returned language 1.0 on those four tickets. Chinese frustration there was 1.54 at 0.31. Eleven calls, 5,635 input tokens, $0.000237 at list price. n = 1 each. TypeSafe's internal 60.0 chart is a different test.
Jev 1.14.0, TypeSafe internal intelligence
60.0 · 58.9 · Please don't trust them
TypeSafe charts posted October 14, 2026, watermarked antibench.com. Overall intelligence: jev 1.14.0 60.0, jev 1.13.0 58.9, kev-27b 46.7, pplx-decider-v1.1-27b 46.0, clef 44.3, clef-flash 38.8, kev-4b 36.9, d1-3b 33.8. Calibration error, lower better: 1.6 and 1.8 for the two Jev rows. Almeida wrote that the measure is an experiment and likely biased toward TypeSafe's models. 60.0 is not JevBench's hosted composite of 71.5. This desk did not rerun the suite.
Jev Day aliases, payouts sentence
jev-1.14.0 · jev-1.13.0 · noul 0.95
This desk, October 15, 2026, Date header 01:22:13 GMT. Two POSTs to https://api.typesafe.ai/v1/systemone, state Help! My payouts have been failing for 3 days. jev-latest returned jev-1.13.0, noul 0.95, billing 0.88 at confidence 0.82, 361 input tokens, desk clock 427 ms. jev-preview returned jev-1.14.0, noul 0.95, billing 0.86 at confidence 0.79, same token counts, desk clock 433 ms. GET /v1/models listed those two aliases. n = 1 each.
Danger-zone image, red square
0.90 on 1.13.0 · 0.39 on 1.14.0 · HTTP 400 without header
This desk, October 15, 2026. A 400 by 400 PNG drawn here, red square from 0.40 to 0.80 of the width and 0.20 to 0.70 of the height. Date header 01:22:14 GMT, jev-preview, no images header, HTTP 400. Date header 01:22:15 GMT, TypeSafe-Danger-Zone-Images: true. jev-1.14.0 Noul 0.39, Box left 0.4, right 0.8, top 0.16, bottom 0.7, confidence 0.93, 760 input tokens, desk clock 457 ms. jev-1.13.0 Noul-only 0.90, 746 input tokens, desk clock 489 ms. A Box on jev-1.13.0 returned HTTP 400. n = 1.
TrueForge weekly client follow-up, 40 requests
37 of 40 · $0.0015 · 16 reviewed
Akshay Pachaar, October 12 and 13, 2026. Both arms 37 of 40 on the right issue and the right status. With Jev, 16 answers went for a second look and review caught 5 wrong matches. Without Jev, 5 went for a second look and review caught none. Printed decision cost $0.0015 with Jev and $0.005 to $0.015 without it. The 150 times line is a separate Opus rate-card estimate; 150 times $0.0015 is $0.225. On October 13, at 09:21 UTC, he wrote that 40 is too small to establish how reliably confidence routes errors to review. On October 14, 2026, Date header 02:34:24 GMT, this desk's jev-1.13.0 call on a written shortlist picked ISSUE-1406 at confidence 1.0, 837 input tokens, desk clock 483 ms. A trap shortlist, Date header 02:35:12 GMT, picked a closed API 504 at 0.55, none at 0.43, confidence 0.51. This desk did not install TrueForge. LangChain's harness and the 500/500 judge eval are other tests.
JevEval, Paris weather Tool Faithfulness
0.254 invented · 0.901 clean
This desk, October 14, 2026, Date header 01:58:11 GMT. Two POSTs to https://api.typesafe.ai/v1/systemone, jev-latest, response jev-1.13.0, HTTP 200. Questions and state follow DeepEval's Tool Faithfulness example. Invented breeze and 40% humidity: Noul 0.03 and 0.63, Score 1.74, Choice stated_it_as_fact at 1.0, weighted mean 0.254, strict_mode 0, 545 input tokens, desk clock 478 ms. Clean 18°C and sunny: Noul 0.82 and 0.97, Score 2.98, Choice dropped at nothing_missing 0.89, weighted mean 0.901, strict_mode 1, 533 input tokens, desk clock 478 ms. The docs walkthrough that prints 0.461 is illustrative. This desk did not install deepeval. LangChain's 500/500 and the cascade paper's 1.7 points are other tests.
SubQ Snap, Stripe ticket
HTTP 200 · noul 0.98 · technical 0.90
This desk, October 13, 2026, about 03:00 UTC. POST https://api.subq.ai/v1/systemone, model subq-snap-preview. Two calls, no image. Urgency noul 0.98, 119 input tokens, 0 output, desk clock 1,195 ms. Three questions chose technical at confidence 0.90, frustration 0.98, noul 0.98, 235 input tokens, desk clock 2,291 ms. No processing-time header. GET /v1/models with the same key listed that one id. The October 12 blog's 61.7 is a different table.
SubQ Snap, Decision Index 0.2.1
61.7 · Jev 57.9 · company table
Subquadratic blog, October 12, 2026. Decision Index 0.2.1 public suite, all requests under about 34K tokens. Snap 61.7, Jev 57.9. The paragraph also prints 61.70 and 57.90. A 16-task enterprise slice averages 67.54 against 61.6 and leads on 13 of 16. Nimble Suite 13 is 82.9% against 76.0%. Decision LCR is 74.0 one-shot against Jev 48.0 chunked. Opened October 13, 2026 UTC, the Decision Index space headers v0.3.2, 149 models, Jev 60.1, updated 2026-10-12. A search of that page for Snap returned no row.
Pairwise Choice, eight Stripe passages
8.86 and 8.07 · other six ≤ 2.42 · one cycle
This desk, October 12, 2026. Two lists of eight passages written here, query How do I rotate a Stripe webhook signing secret?, POST /v1/systemone, jev-latest, response jev-1.13.0. On the Stripe shortlist a 0-9 Score put Roll secret at 8.86 and the signing-secret page at 8.07. The other six scored 2.42 or below. Spearman against this desk's intended order: numeric 0.690, four-level Score 0.738, Noul 0.357, pairwise Bradley-Terry 0.548. Pairwise ranked rotating the API key third. The majority graph cycled A, F, E. Four calls, 9,664 input tokens, $0.000406 at list price. These texts are not a retrieval corpus.
Quyet-1.0-Large, frozen JevBench v1.6.0
71.39 · Capability 81.7 · rank 1 of 92
quyet.ai, opened October 12, 2026. Frozen release v1.6.0, 92 systems. Quyet-1.0-Large official score 71.39, Capability 81.7, Intelligence 73.4, Calibration 90.0, $0.045 per 1,000. Jev 1.13 on that table is 71.09 and Capability 76.5. The October 8 open-weights rank line still says Capability 81.7 and official composite 3. H2O's October 7 composite for Quyet is 71.4. We did not reopen the frozen file, and we did not run the weights.
Blink v0.3, Decision Index 0.2.1
57.48 · Jev 57.91 · self-run
PixilabAI/Blink-v0.3-26B-A4B-NVFP4, opened October 12, 2026. The card's own run of Decision Index 0.2.1 is 57.48. The Jev cell it copies is 57.91. Surogate Rune v3 on that table is 57.44. The card says the run is not a board submission. Calibration error is 0.016 over 216,942 decisions, at temperature 1.3. We did not run the weights.
JevBench API board, October 11
Sage 74.0 · Jev 71.5 · Microsoft-Decision-1 69.1 · 27 systems
Benchmark Heaven API board, opened October 11, 2026. Release v1.6.1, board revision v1.7.44, 27 hosted systems. Sage 1.3.0 is first at composite 74.0, capability 78.6, median 0.15 s. Jev 1.13.0 is fourth at 71.5, capability still 77.1. Microsoft-Decision-1 is sixth at 69.1. The v1.7.31 note prints Capability 70.78, Calibration 84.40, Composite A 69.11, and a native median of 459 ms. Mercury Decide stays third at 72.4. The October 8 reading ranked 26 systems and had no Microsoft row. We did not rerun the suite.
H2O-Lightning-4B v1.1, October 7 composite
72.5 · Jev 71.5 · revision v1.7.17
Jonathan McKinney, H2O blog, October 7, 2026. Official composite, 104 open-weight systems, board revision v1.7.17. H2O-Lightning-4B v1.1 is 72.5. Jev 1.13.0 is the unranked reference at 71.5. Intelligence on the H2O row is 60.0, calibration 90.0, estimated $0.021 per 1,000 decisions. Raw median 0.029 s. Adjusted median 0.21 s. The blog's 205 of 231 is the public-item adapter count, a different print. The October 8 open-weights card, capability 75.0, is a later reading. We did not run the weights.
Drex v1.5, Decision Index 0.3.1 public
58.08 · Jev 57.96 · public index
nace-ai/drex-v1.5, opened October 11, 2026. Public index only, edition 0.3.1. Drex v1.5 is 58.08. The card's Jev 1.13.0 cell is 57.96. Nimble 9B v3 on that table is 57.19. The card says the three sit inside a 0.9-point tie band. The September 30 chart's 58.28 is edition 0.2.1. Drex DLM's 52.31 is edition 0.2. The card's 199 of 231 reprints the September JevBench public count. We did not run the weights.
OpenBles Firewall 1, agent test split
119 of 120 · 0 benign look-alikes flagged
stas4000/openbles-firewall README, opened October 11, 2026. Agent test split, 120 attacks and 120 benign look-alikes. OpenBles caught 119 and flagged none of the look-alikes. Jev 1.13.0 on that split, run October 2, flags 9.2%, which is 11 of 120. The classic split is a different set: 55 of 60 against Jev at 33 of 60. We did not run the weights.
Decision 3.0 d3, index 0.3.1
64.7 · public 65.5 · highlight 65.45
vllm-sr/d3, opened October 11, 2026. The card's own Decision Index 0.3.1 cell is 64.7, public 65.5. The highlight prints public-suite 65.45 for edition 0.3, and +8.5 against a Decision 2.0 board cell of 56.97. The same table's Decision 2.0 row is 55.9. Jev on that table is 60.1. d3-nano's table prints 37.2, under Liquid d1-3B at 40.1. The card calls its own row an internal evaluation. We did not run the weights.
Mizan 4B, Persian mean
78.7 · Jev 83.7 · 3,000 records
xmanii/Mizan-4B, opened October 11, 2026. Mean of seven public Persian tasks, 3,000 records. Mizan 4B is 78.7. The Q4 GGUF is 78.3. Clef-flash is 83.0, Clef 27B is 83.4, Jev is 83.7, and Laya is 62.4. English held-out BANKING77 77.0 and the synthetic Persian 91.2 are other prints. We did not run the weights.
THX-01, support-ticket average
98.4 · Jev 97.4 · 2,843 tickets
doofz/THX-01, opened October 11, 2026. Four sets, 2,843 tickets, fifteen categories. THX-01 averages 98.4, ECE 0.003. Jev 1.13 averages 97.4, ECE 0.007. Number lookup on a different set is 93.4, and Jev on that line is 96.6. Banking77 in the 39-task suite is 65.9. We did not call the API.
Clef-omni, October 9 ten-row table
BANKING77 macro-F1 94.8
Michelle Chen, Cloudflare blog, October 9, 2026. Clef-omni's BANKING77 cell is macro-F1 94.8, against Clef 94.20, Clef-flash 90.93, and Jev 79.74. CLINC150+OOS is 97.7. When2Call is 63.3, and Jev on that row is 80.97. A 21-second video with sound is about 1.5 seconds, a different print from these rows. The October 1 median of 209.3 ms is a different print again. Price on the post is $0.15 per million input tokens. We did not call the API.
PurpleMIST-Flash-1.0, African Typed Decisions
0.668 accuracy · KL 0.214 · 15,050 decisions
Olaverse card, opened October 10, 2026. Translated set, eight languages, 3,010 cases, 15,050 decisions. PurpleMIST-Flash-1.0 accuracy 0.668, KL 0.214, Brier 0.115. DeepSeek-V4-Pro on the same harness is 0.669 and KL 1.125. English typed-decisions, 2,000 decisions, is 0.720, and the card's Jev cell there is 0.727. A self-run of Decision Index edition 0.3 prints public index 46.7. Those three numbers are different tests. Text only. We did not run the weights.
vegaml, 800 phishing emails
75.4% · Jev 61.9% · 252 of 400 caught
Nandha Kishor M, vegaml README, opened October 10, 2026. Phishing screen, 800 emails, Vega 75.4% accuracy, 252 of 400 caught, recall 0.63. Jev 1.13.0 on the same items is 61.9%, 99 of 400, recall 0.25. McNemar p = 3e-11. The README says this table is the rows where Vega leads. The 0.763 adapter score is 2,050 LocalLLaMA/typed-decisions items, a different test. Decision 2.0's Vega at 56.5 is a different checkpoint. We did not run the weights.
TypeLLM, 231 public JevBench tasks
228 of 231 thinking · 195 of 231 without · hard 109 of 111
TypeLLM, September 23, 2026. One checkpoint, RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead, two settings, 231 public tasks. Thinking is 228 correct. No thinking is 195. Hard moves from 76 of 111 to 109 of 111. Easy moves from 48 of 48 to 47 of 48. Original moves from 71 of 72 to 72 of 72. The thinking bin from 0.9 to 1.0 holds 225 tasks, 222 correct. The homepage prints 98.70% and 84.42%. It also prints Jev 1.13.0 at 86.58%, and the eval README attributes that row to Open-Jev. The hosted name typellm-latest is not this checkpoint. We did not rerun it.
Microsoft-Decision-1, Microsoft's October 9 comparison
4.5× Quyet-1.0-Large · 35× GPT-6 Sol · 1.3% flips
Achint Srivastava, Command Line, October 9, 2026. Microsoft's own 36-benchmark comparison, nearly 150,000 questions. The post says Microsoft-Decision-1 had the highest accuracy and does not print it. Speed: 4.5 times quicker than Quyet-1.0-Large, 35 times quicker than GPT-6 Sol, and P50 about 35 times faster than GPT-6 Sol. Eight perturbations flipped the decision on 1.3% of cases on average. The appendix does not name Jev. The Foundry benchmarks tab, opened October 10, prints no accuracy percentage. JevBench's API page that day, release v1.6.1 and board v1.7.29, had no Microsoft-Decision-1 string. We did not rerun it.
Bag of decisions, WANDS and ESCI
WANDS mean 0.574 to 0.610 · median 0.561
Doug Turnbull, October 8, 2026. CSV at commit 8734fde. WANDS, the note's default of 480 queries: one Noul mean NDCG@10 0.574262, a GPT-5 list of Nouls 0.609814. Both medians are the float 0.5609447680982702. The score is retrieval plus 10 times the summed probabilities, top 100, model jev/jev-latest. The printed BM25 row uses title boost 9.3 and description 4.1. The two Jev rows use title^9.4 and description^4. ESCI, default 1,000 queries, means 0.313621 and 0.360079. cheat-at-search discounts by 1/rank and ideals a perfect top 10. The blog prints 0.574 and 0.610. We checked the CSV and did not rerun the searches.
Step-Jev, BrowseComp-Plus search agent
19.6% with steps · 5.4% final only · 15.6% untrained
laxman, October 8, 2026. Qwen3-4B on BrowseComp-Plus, Dr. GRPO, one seed. The reward that grades the final answer and every step scores 19.6%. The final answer alone scores 5.4%, with 10.5 tool calls and 15% repeats. No training is 15.6%. A string-match answer key is 22.9%. The paired gap for adding steps is +14.2 points, interval +8.1 to +20.4. Hosted Jev, zero-shot, is 0% on that page. On October 9 this desk scored the repository's own test states with jev-1.13.0. The no-step state was incorrect at 1.0. The five-step fixture was correct at 0.05. A Moon-landing question written here scored a no-search answer of 1965 incorrect at 1.0, and a repeated NASA search stayed positive at 0.53. real_jev_gate.py's rollout files are not in the repository. We did not rerun the table.
Arena, typesafe/jev-router, October 7
+3.6% net · $0.155 median per task
Arena, October 7, 2026, 22:29 UTC. More than 4,700 agent sessions. The thread says similar performance to DeepSeek V4.1 Flash (Max) costs 38% more, with median model request latency 1.7 times higher. Printed clocks are 6.18 seconds against 3.64 seconds, and P90 30.59 seconds against 14.26 seconds. The mix is DeepSeek V4.1 Flash 28.1%, GPT-6 Astra 23.5%, Sol 22.8%, Luna 11.1%. Steerability is +10.0% against DeepSeek at -0.2%, and +10.48 for Claude Opus 5.5 (High). The September 25 count of 237 of 423 is a different test. We did not call the router.
Decision Index 0.3, October 7
v1.1 62.8 · Jev 60.1 · GLiDE no-thinking 60.2
Opened October 8, 2026, the Decision Index space still headers v0.3, 115 models, Jev jev-1.13.0, updated 2026-10-07. The head-to-head panel still lists Perplexity Decider v1.1 at 62.8, Fastino GLiDE no-thinking at 60.2, and Torchcast Decision 27B at 59.9, with Jev as a reference at 60.1. Hugging Face Decision Index space, opened October 7, 2026. Header: v0.3, 112 models, Jev jev-1.13.0, updated 2026-10-06. Full score: Perplexity Decider v1.1 rank 1 at 62.8, Fastino GLiDE no-thinking rank 2 at 60.2, Torchcast Decision 27B rank 3 at 59.9. Jev 1.13.0 is a reference at 60.1. The v1.1 area cells on that row are 45.8, 62.3, 64.7, 80.6, and 51.4. The v1.1 card's own table prints 61.56 against Jev at 57.9. A search for Drex returned no row. We did not rerun it.
HiringCafe Spearman, October 7
Query 0.74 / 0.71 · resume 0.71 / 0.67
Hamed Nilforoshan, October 7, 2026. Query-to-job Spearman is Jev 0.74, OpenAI 0.71, Gemini 3.1 FL 0.63. Resume-to-job is 0.71, 0.67, and 0.61. The post's summary line says the OpenAI call is twice as expensive and 5 to 10 percent worse. The thread does not print the pair count, a price, or a latency. It says HiringCafe has 2.5 million users. We did not rerun it.
Drex DLM, Decision Index 0.2
52.31 · kit Jev 51.67
Nace, October 6, card opened October 7. The post calls the weights Drex 1.1. The card title is Drex DLM. Decision Index 0.2 prints 52.31 against the kit's Jev cell of 51.67. Weights are CC BY-NC 4.0. The call is POST /v1/systemone. The hosted Drex 1.5 row on edition 0.2.1 is 58.28. A search of the 0.3 space for Drex returned no row. We did not run the weights.
PolicyLM-1.7B, custom policy
0.842 at a 0.5 cutoff
Musubi Labs card, opened October 7, 2026. Custom-policy accuracy 0.842 at a 0.5 cutoff. Policy edits followed 0.528. H100 median 22 ms. gpt-oss-safeguard-20B is 0.909. The card says every set except OR-Bench was also used in development. The table has no Jev column. The model returns a score from 0 to 1 for each category, up to 16 in one pass. We did not rerun it.
JevBench API board, October 7
Jev 71.5 · rank 3 · capability 77.1
Opened October 8, 2026, the API board is still release v1.6.1, revision v1.7.21, and ranks 26 hosted systems. Jev 1.13.0 is fourth at 71.5. Capability is still 77.1. Mercury Decide is third at 72.4. OpenAI Decisions is seventh at 62.5. Fastino GLiDE is fourteenth at 30.6. Imajev-4B on the open-weights list is capability rank 31 at 61.9. Benchmark Heaven, opened October 7, 2026. The API page says release v1.6.1 and board revision v1.7.12. The revision note says headline scores, Capability, Composite, and every rank are unchanged. Composite rank is Sage 1.3.0 at 74.0, Liquid d1 at 73.0, Jev 1.13.0 at 71.5, wity-1 at 70.8, and OpenAI Decisions at 62.5, interval 51.6 to 64.6. Jev's capability on that board is 77.1, Intelligence 63.6, Calibration 90.6, $0.032 per 1,000 decisions, p50 0.24 s, capability rank 2. d1 answered all 1,500 items and is capability 74.4. OpenAI Decisions, gpt-6-luna, is equated from 598 of 600 items and is capability 73.5. GLiDE is composite 30.6, rank 11, at $0.137 per 1,000, and capability 77.0 outside the cost cap. On the open-weights board Jev is an unranked reference at the same 77.1 and 71.5. Imajev-4B is capability rank 17 at 61.9. 72.1 stays the v1.5 composite. We did not rerun it.
Tev1, Ollama's 3,880 decisions
Tev1 4B 73.3% · 0.8B 63.5% · Jev 76.0%
Ollama's tev1 page, opened October 7, 2026. Mean on 13 public datasets, 3,880 human-labeled decisions. Tev1 4B is 73.3%. Tev1 0.8B is 63.5%. Nimble 9B is 75.7%. The Jev cell is 76.0%, and the page says that cell is Bespoke's published API run. Together's own development set, 880 of 1,000, is a different test. The Together page opened the same day lists $0.04 per million input tokens. The September 23 post said $0.042. We did not run it.
fsdecide, ForkScape's 450 items
Review 91.3% · Jev 56.3% · general set Jev 90.5%
ForkScape, October 5. 149M ModernBERT. Review is 80 items: fsdecide 91.3% (83.0% to 95.7%), jev-latest 56.3% (45.3% to 66.6%), 35 of Jev's errors are finished work called fail. Command intent is 100% against 78.3% on 60 items. Browse intervals overlap. Unsafe allows are 0 of 85 for Jev and 1 of 85 for fsdecide. The authors' 1,248-request set, which the post labels JevBench, is jev-latest 90.5% and fsdecide 60.5%. A model that ignores its input scores 89.4 on that file's coherence column. Not Florian's board. jev-latest's bill on the 450 items is $0.0119. We did not rerun it.
Decision 2.0, Vega index cell
Index 56.5 · JevArena 74.0 · median 71.4 ms
Xunzhuo Liu, October 3. Six Apache-2.0 cards, opened October 5. Vega, 29.37B parameters, prints Decision Index 56.5, JevArena 74.0, transfer 58.7, and a single-question median of 71.4 ms. The card calls 74.0 level with AutoJev-27B at 72.1 and does not print the interval. No card prints a Jev row. The September 28 board's Jev cell is 57.91. Kai's index cell moves from 6.5 to 16.3 against Decision 1.0. The vLLM post says 64 questions in one pass. The cards do not print 64. Kai's config prints max_options 255. We did not run the weights.
Vela 2.0, 9B index cell
41.63 calibrated · 41.09 default · Lux harness 46.23
Xunzhuo Liu and vLLM, October 7. Cards and the 9B evaluation file opened October 8. Vela-2.0-9B prints Decision Index 0.2.1 at 41.63 with noul_calibration on, and 41.09 with it off. The file prints Decision-2.0-Lux-9B at 46.23 in that column and does not print a Jev row. The blog says the default keeps 89% of that base. The October 5 Lux card prints 46.3. The same file says its harness reproduces the released cards within 0.1 and prints Nox at 42.55. Nox's October 5 card prints 43.8. RAGTruth on the file is 0.774 against the LettuceDetect v2 mmBERT encoder at 0.743, and the generative qwen-2b row is 0.817. We did not run the weights.
JEV-27B-VL, pick and place
15 of 20 scenes · six-benchmark mean 84.07
AutoTrust, October 4. The card's October 3 section prints 75% of 20 pick-and-place scenes, 15 of 15 grasped cubes in the tray, and misses 3.0 to 3.7 cm off the cube. Direct 8-way motor commands completed 0 of 10. Browser tasks with element text are 95% of 60. Numbered boxes alone are 10%. A September 27 six-benchmark mean is 84.07 against hosted Jev at 83.85. The JevBench column in that mean, 88.70 against 87.18, is not Florian's board. On 800 labelled items Jev's published cells stay ahead at every option count from 2 to 16. The post's Space Invaders line, 40 of 40, is not in the October 3 section. We did not run it.
basal-1.5, Werdykt v1
72.12% · Jev 81.64% · p50 33.8 ms
Remek Kinas, October 4. Hidden set, 5,000 decisions, 10 categories in Polish and in English. JSON export the same day: basal-1.5 72.12% (71.02% to 73.26%), p50 33.8 ms, $0.0477 per 1,000, SGLang with the context limit raised to 32,768. Jev 1.13.0 is 81.64% (80.70% to 82.58%), p50 286.4 ms, $0.0710 per 1,000, API, no protocol deviation on that row. max is 77.28% at 67.1 ms. mini is 60.66% at 14.2 ms. A closed set of 3,000 new Polish decisions is 93.1% against basal-1.0 at 87.4%, interval on the gap +4.5 to +6.8. The site schedules the weights for October 5, 09:00 Polish time. On October 4 the v1.5.0 tag returned 404. Opened October 8, 2026, GitHub main titles the README basal 1.5. We did not run it.
StartLux-Decision-27B, Decision Index 0.2.1 self-run
63.88 · board Jev 57.91 · 31 of 38
StartLux, October 3. README scores the 27B at 63.88, the 35B-A3B at 61.55, and the 9B at 58.63, with the board's kit. The Jev cell it cites is the September 28 board's 57.91. The README says the runs are not on the board. Knowledge and Reasoning is 44.3 against Jev at 51.4. GPQA Diamond is 34.0 against 71.4. Fourteen of the 38 benchmarks had their public train splits in the training data. Public JevBench items in the Intern-Decision bundle are 208 of 231, against their Jev cell of 199. The README says that count is not the official JevBench score. Three-question latency on one H200 is 102.3 ms for the 27B and 26.0 ms for the 4B. The same request's TypeSafe server time is 64.0 ms, mean of 100. Code is Apache-2.0. Weights are CC BY-NC 4.0. We did not run it.
Perplexity FireRed, one Elite Four run
592 ms median · 137 calls · $0.028
Aravind Srinivas, October 4, 10:10 UTC. The post says Pokémon FireRed's Elite Four and Champion were cleared in one shot on Perplexity's Decisions API. Median API response 592 ms, p95 987 ms, 96.4% of responses under one second, estimated inference cost $0.028, 137 live calls. The post does not say whether the median is client time or server time. It prints no Jev cell. The attachment is a video. This is not the 7,210-row panel. We did not rerun the game.
jevmod, sexual labels on OpenAI's moderation eval
AUROC 0.984 · F1 0.866 at 0.80
Omar Hernandez, remeasured September 23. 1,680 messages, human sexual label mapped to nsfw, base rate 0.15. AUROC 0.984, F1 0.866 at the shipped 0.80 line, best F1 0.877 at 0.85. Llama Guard 3 8B is 0.843. ShieldGemma 2B is 0.968. ECE 0.083. The $0.044 per 1,000 messages is the whole seven-category request. ZeroNSFW scores pictures on the host and prints no adult-image accuracy. We did not rerun either.
NIRNAY 450M, Banking77 phase_b JSON
0.8792 · n=3080 · fitted ECE 0.045
Gautam Kishore, October 3. phase_b accuracy 0.8792, Brier 0.208, ECE raw 0.089, fitted 0.045, on 3,080 Banking77 test rows. The October 3 headline 0.8656 is the phase_a file. The card prints Jev 1.13.0 at 0.803 and also says that column was not measured in the repo. The post prints 0.64 and does not link it. The JSON files have no Jev field. Local card timing is 209 ms on M4 and 361 ms on CPU. A reply lists 310 to 478 ms locally. JevBench public on the card is 0.550 on 231 items, hard 0.396. We did not run it.
WaterSheep, in-distribution and held-out
77.8% · ECE 0.026 · held-out 61.2%
Samrat Dutta, October 3. Apache-2.0 fine-tune of ModernBERT-base. POST /v1/systemone on port 8766, plus a multi-label type. In-distribution 77.8%, ECE 0.026. Held-out datasets 61.2%, ECE 0.043. Those two rows do not print a question count. GoEmotions is 22.4% of 2,000. HateCheck is 75.1% of 2,000. There is no Jev column. We did not run it.
GEV-26B-Decide, self-run of Decision Index 0.2.1
adaptive 62.48 · board Jev 57.91
AutoTrust, October 3. Gemma 4 26B, about 4B active, same weights as JEV-Gemma4-26B-A4B. Adapter, head, and calibration are Apache-2.0. The card's own scoring prints adaptive 62.48, raw 70.66, breadth 62.00. Knowledge uses adaptive thinking, 33,856 requests. The other four areas are System 1 from 150,759 requests. Adaptive median on Knowledge is 13.4 seconds, so that mode misses the kit's 1,000 ms cap. A 1,754-row holdout goes from 73.3% to 83.4%. The post's four puzzles go from 21% to 86%, 54% to 99.5%, 52% to 100%, and 76% to 99.5%. Not a row in the September 28 board file. We did not run the model.
GLiDE, October 1 self-run of Decision Index 0.2.1
64.81 · Jev 57.91 · 31 of 38
Fastino, October 1 blog, chart October 3. GLiDE 64.81 against the published Jev cell of 57.91. The blog says all five areas and 31 of 38 benchmarks. Tools 83.5 against 75.1. Knowledge 62.9 against 51.4. CLadder 88.7% against 72.6%. CRUXEval 92.6% against 73.0%. The October 3 chart also prints Clef 61.2, Clef-flash 57.1, and pplx-decider 56.39. The caption says the Cloudflare scores are self-reported and that Fastino ran Perplexity. The axis starts at 56. Submissions for 0.2.1 are paused. We did not call the API.
pplx-decider, 7,210-row panel
85.71% · Jev 84.51% · Qwen 74.76%
Perplexity, October 1. Eleven benchmarks, September 2026. Jev leads six rows. pplx-decider leads four. ContractNLI ties Qwen3.8-27B and pplx-decider at 80.78%. Public-hard JevBench on this chart is 101 rows, Jev 73.27%. The October 1 quickstart priced input at $0.04 per million tokens. The October 7 quickstart prices the API at $0.02. A Fastino chart prints 56.39 for this model on Decision Index 0.2.1, a different suite. Hendrickson's October 3 note reports an EXL3 build at 15.35 GB. We did not rerun the panel.
StreamDecisionBench, log-AUC over 0.5 to 8 seconds
Kev-27B 62.15 · Jev API 59.63
Jacob Lin, October 2. The chart image is the table. The live page HTML did not render it. Kev-27B 62.15, Jev 59.63, GPT-5.6-Terra none 54.11, Clef 30.96, Clef Flash 19.69, Nimble-9B 14.43, Laya English 0.42. The paper's October 1 table is a different scoring. Nimble's 14.43 is not the 292 of 324 holdout. We did not rerun the streams.
CUA Speedrun, OSWorld-50 callout
Jev 9.8% · 125 seconds a task
Pranjal Aggarwal, October 2. The chart callout is 9.8% and 125 seconds. The chart annotation says as fast as Astra xhigh, 9x lower accuracy. The opening post says 1.5x slower than Opus-5.5-xHigh and 10x lower accuracy. The thread puts a Jev call under 0.2 seconds and a screen parser near 500 ms a step. Other dots on the plot have no printed coordinates. We did not rerun the 50 tasks.
Open-Jev 2B, one 64-step pilot
84/128 to 99/128 · OOD 83/128 to 95/128
Zefan Cai, October 2. Released Open-Jev-2B, frozen backbone, LoRA plus head, one H200, 256 train rows. Synthetic controls, not JevBench. Test NLL 0.8900 to 0.6046. ECE 0.1353 to 0.1088. Timeline out of distribution falls from 58.3% to 54.2%. The 231-task card, 197/231, is unchanged. We did not rerun the pilot.
37 datasets, abstract only
346,009 requests · under $10 · 27 of 37
Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa, arXiv:2609.37647, submitted September 29. jev-1.13.0, one frozen template per dataset. Beats Qwen3.8-27B on 27 of 37. None of Qwen's nine leads sits outside the bootstrap intervals. Beats Gemma-4-E4B on all 37. Belebele 86.7% across 122 languages. UNFAIR-ToS micro-F1 0.50 to 0.75 after thresholds tuned on training data. We read the abstract, not the HTML or the PDF.
WrangleBench, perfect matches per 200
137.3 with Jev · 105.1 without
John Crown, Wrangle, September 30. The chart's axis is perfect matches per 200. With Jev the bar is 137.3. Without Jev it is 105.1. The post calls that a 30.64 percent gain, at 75 percent of the previous reranking cost. The chart prints no price, no query count, and no name for the previous ranker. Crown says the full benchmark is still to come. We did not rerun the searches.
MMAI Gym, five CYP inhibition assays
12,563 molecules · Jev tops out at 0.69
Mikołaj Mizera, October 2, chart credited to Insilico Medicine. A gradient-boosting ensemble with no calibration step, against Jev 1.13. AUROC is 0.94, 0.89, 0.89, 0.89, 0.90 against 0.76, 0.73, 0.60, 0.78, 0.67 on 1A2, 2C9, 2C19, 2D6, and 3A4. Balanced accuracy is 0.87, 0.79, 0.83, 0.74, 0.81 against 0.65, 0.55, 0.49, 0.67, 0.57. Jev's calibration curves stop at 0.69. The post links no paper and no repository. We did not rerun the assays.
You.com risk sweeps, invoice and a blind read
$0.43 for three · means 4.25 and 4.25
Edward Irby, October 1. MIT repo youdotcom-oss/risk-analysis-server. Three escalated sweeps: Qwen $0.0619, You.com $0.37, Jev $0.0025, total $0.43, or $0.14 each. At a 0.7 threshold, Gulf scored 0.58 and stopped after the 0.5 arm had marked it critical at 0.56. Qwen escalated 6 of 11 attempts. Jev escalated 9 of 9. Blind means are both 4.25. Qwen's Gulf triage read 0.35, then 0.68, then 0.50. We did not run a sweep.
Bud Decision Studio, GB10 training table
Jev-Omni 62% to 77% · CLM 8B 39% to 68%
Bud Ecosystem, October 2. Local server at 127.0.0.1:8420, POST /v1/systemone. Eleven open models. The README's GB10 table does not print an example count. Jev-Omni on business workflows goes from 62 percent to 77 percent, general-decisions delta 0.0. CLM 8B goes from 39 percent to 68 percent, +11.2. Laya Multilingual goes from 34 percent to 62 percent, +12.7. GitHub lists no license. We did not install the app.
DSPy office-hours chart, Jev to Luna
Luna 92.5% · live 88.0% at τ 0.78
Nathan LeClaire's September 30 thread, from the DSPy office hours. The chart is 100 test pairs, policy frozen on 30 validation pairs. Standalone Luna is 92.5 percent at $0.746 per 1,000. A live threshold near 0.78 is 88.0 percent at $0.583 per 1,000. The Jev-only end has no numeric callout. DSPy 3.4.0, September 25, adds the TypeSafe client and prints no accuracy table. The patterns repo pins jev-1.13.0 and sets the shell gate at 0.2 and 0.6. We did not run the demos.
Strands Decider 2B, 167 of 231 public tasks
167/231 · median 115 ms · 3rd of 33 in the 2B class
Strands Labs, October 1. Apache 2.0, 1.9 billion parameters, Qwen3.5-2B-Base, a pointer head, and a rank-16 LoRA. The checkpoint is StrandsAgents/strands-decider-2B-hobson-v19. At the 3072-token window, v19 is 167 of 231, Brier 0.342, ECE 0.052. Easy, standard, and hard are 1.000, 0.875, and 0.505. An RTX 3090 median is 115 ms and the 95th percentile is 299 ms. The blog's opening line says tens of milliseconds. The rank is the September 25 v1.4.2 board, 89 systems: 50th overall, 3rd of 33 in the 2B class. The file says this is public-task accuracy and that sealed items are absent. On their harness, decider-2b v11 is 175 of 231. With the state and the options fixed, a changed question gets the same answer about 94% of the time. temporal_numeric is 0.267. We did not run the model.
HydroJEV, simulated water-network alarms
macro-F1 0.63 · gate 69/200 · 76/200
Tianwei Mu and coauthors, arXiv:2610.02048, submitted October 1. jev-1.13.0 on stochastic EPANET windows. In-distribution macro-F1 is 0.620, 0.628, 0.628, and 0.644, mean 0.63, against a hand-written rule tree at mean 0.59. No single gap against that tree was significant. A classifier trained on 200 labels of the same subtypes is mean 0.91. Leave-one-subtype gaps are +0.361, +0.387, +0.390, and +0.417. Median compute on 2,722 calls is 1.215 seconds. An agreement gate settles 69 of 200 and 76 of 200 fresh windows. On the look-alike set, 32 of 76 accepts were wrong, including 21 attacks. A glm-5.2 cascade scores 0.745 and 0.565 against 0.735 and 0.549 for glm-5.2 alone. The windows are simulated. We did not rerun the calls.
DimABSA, ten-corpus valence and arousal
RMSE 1.0645 · continuous F1 52.09 · 44.06
Yiqun Zhang and coauthors, arXiv:2609.35293, submitted September 28. Frozen jev-1.13.0 plus 488 coefficients fit on CPU. Task 1 micro RMSE is 1.0645, 0.0018 under PAI at 1.0663, among 14 teams that report all ten corpora. The paper says that margin cannot be tested without the other systems' predictions. Task 2 continuous F1 is 52.09, between sixth and seventh of 12. Task 3 is 44.06, between fifth and sixth of 9. Raw RMSE before the ridge fit is 2.0731. Llama-3.3-70B in the same table is 2.5683, 46.40, and 38.62. We did not rerun the scorer.
jev-doc-search, two annual reports
2 questions · both marked correct
VectifyAI, October 2. Apache 2.0. jev-1.13.0 walks a PageIndex tree. A flat Choice stops at 32k tokens and at 255 options. NVIDIA's fiscal-2026 10-K is given as 93 pages and about 76k tokens. Citigroup's 2025 10-K is given as 318 pages. The printed table has two questions. Both are marked correct: NVIDIA total revenue on page 51, Citigroup net income on page 134. We did not run the scripts.
Clef and Clef-flash, Cloudflare's published tables
median 209.3 ms · 38.8 ms · Jev 524.1 ms
Cloudflare, October 1. Apache 2.0 weights, hosted as @cf/cloudflare/clef at $0.24 per million input tokens and @cf/cloudflare/clef-flash at $0.09. The blog's ten-row table has Clef or Clef-flash higher on seven rows. Jev is higher on When2Call, 80.97 against 72.37 and 65.58, and on BRIGHT, 47.52 against 45.91 and 39.26. DiffusionGemma Jev leads PhishNChips at 85.35. Invoice exact actions: Clef 64.7, Clef-flash 57.1, Jev 61.8. Agent trace: Jev 71.6. The model card's GPQA Diamond row is Jev 78.3 against 48.0 and 51.0. RAGTruth for Clef-flash is 35.6 against Jev 76.5. The card calls this an internal Decision Index 0.2.1 run. It does not print the 57.91 composite. We did not open the live demo and we did not call the endpoint.
Open d1, d1-3B card, Decision Index 0.2.1
d1-3B 48.57 · d1-omni-600M 15.95 · no Jev row
Liquid AI, October 7, 2026. Cards opened October 8. d1-3B prints 48.57 on Decision Index 0.2.1, official scorer, not a leaderboard submission. Areas: Knowledge 23.8, Language 56.4, Retrieval 52.8, Tools 74.5, Arts 36.3. The same table prints Winnow-12B at 50.02 and Decider 35B-A3B at 47.11. d1-omni-600M prints 15.95. A seven-benchmark mean is 82.9. An eight-benchmark mean that adds HelpSteer2 at 36.7 is 77.1. The hosted chart's 58.9 and the JevBench composite 73.0 are other tests. License file is LFM Open License v1.0, commercial use under $10,000,000 annual revenue. We did not run the weights.
Liquid d1, internal Decision Index 0.2.1
d1 58.9 · Jev 57.9 · +1.0
Liquid AI, September 29. The chart is captioned as an internal reproduction of Decision Index 0.2.1. Five areas: Arts 45.5 against 37.7, language 67.6 against 62.0, retrieval 60.7 against 55.4, tools 74.1 against 75.1, knowledge 43.3 against 51.3. Counts are 7, 10, 6, 5, and 10. The public kit's tie band is 0.25. The docs name model d1:free and POST https://api.liquid.ai/decisions/v1/systemone. Output tokens are always 0. No context window, parameter count, paid price, or license in the sections we read. We did not re-download the September 28 board file, and we did not search it for d1. We did not call the endpoint.
Drex 1.5, 199 of 231 against Jev's 201
199/231 · Jev 201/231 · hard 82/111 tied
Nace, September 30. The post calls Drex 1.5 first on Decision Index 0.2.1 and on JevBench, at $0.04 per million input tokens, sub-second, 128k context, 250 million free tokens. The chart on the next post prints JevBench 86.2%, 199/231, against Jev at 201/231, and a hard-tier tie at 82/111. Decision Index on that still is 58.28, +0.37 over 57.91. Long context, 32K-128K, is 93.4% at a median of 2.0 seconds. The product page says under 10B parameters. The $0.04 line is the post. The page we opened did not print it. An October 1 search of Benchmark Heaven for Drex 1 returned no match. We did not rerun it.
Rox rerank, relevance kept at 200 chunks
Jev-Score 96% · production 84%
Rox, September 28. Mean over 75 queries. At 200 chunks the printed callouts are Jev-Score 96% and production, labelled GPT-5 Mini, 84%. GPT-6 Astra is drawn along the top and has no printed number. The post, quoted by TypeSafe on September 30, says 20 times faster, 10 times cheaper, and 12% more accurate. The multiples are not on the chart. We did not open a methods note.
Jev Sentinel, two cost estimates
TechCrunch $2.94 · demo about $55 / million
TechCrunch, September 30, Tim Fernholz. One monitoring job, in theory, $2.94 with Jev against $372 with a frontier LLM. The QueryStory demo page does not print those two figures. It prints about $55 per million actions with Jev against about $6,900 for a frontier judge, a 0.23 second median, about 70 checks a second from one laptop, and 0 refusals on 53,870 attack payloads. deepset/prompt-injections, 116 texts, AUC 0.98. We did not open the GitHub repo.
Mercury Decide, no score on the post
14 / second · no JevBench cell
Opened October 8, 2026, the JevBench API board ranks Mercury Decide third at composite 72.4. Inception, September 30. The post calls Mercury Decide the most intelligent decision model on OpenRouter, citing JevBench v1.4, at up to 14 decisions a second. It prints no score. Opened October 8, 2026, the free model page returned, priced Free, with up to 14 decisions a second and no JevBench score. The free model URL returned Not Found. OpenRouter's Mercury 2.5 page describes a System One endpoint, output tokens free, the same /v1/systemone schema as Jev. An October 1 search of Benchmark Heaven for Mercury Decide returned no match. We did not rerun the board.
October 1, JevBench text has no Mercury Decide or Drex 1
still v1.5.4 · 106 systems
The same Benchmark Heaven URL, opened October 1, still says v1.5.4 and 106 ranked systems. Cygnet 73.7, Winnow-12B Q8 73.2, and Jev 1.13.0 72.1 were not recopied. A search of that page for Mercury Decide and for Drex 1 returned no match. We did not rerun it.
JevBench v1.5, Cygnet and Winnow tied at the top
Cygnet 73.7 · Winnow 73.2 · Jev 72.1
Benchmark Heaven, v1.5.0, posted September 28 and opened September 29. 1,624 decisions: 904 open, 601 of them published, plus 720 sealed. Sealed is half of Intelligence. Official A is an equal-weight harmonic mean. Cygnet 73.7 (71.1 / 87.0 / 91.0 / 56.4, about $0.028, p50 0.23 s). Winnow-12B Q8 73.2 (74.4 / 84.1 / 86.1 / 56.6, about $0.028, p50 0.34 s). The page calls that pair a statistical tie. Jev 1.13.0 is 72.1, official rank 3, capability 80.0 and first among 56 Jev-class systems. 89 of 103 systems are ranked. 75 of 88 adjacent pairs are ties. Imajev-4B's addendum would place sixth at 70.4. Plumb-4B would place fourth at 71.6. Neither is on the frozen ranked list. 63.29 is the v1.4.2.2 cell. We did not rerun it.
JevBench v1.5.4, the addenda get numbers
v1.5.4: 106 ranked · Imajev #9 70.4
The same Benchmark Heaven URL, opened September 30, says v1.5.4 and numbers 106 systems. Cygnet 73.7, Winnow-12B Q8 73.2, and Jev 1.13.0 72.1 stay on top. The page still calls Cygnet and Winnow joint leaders. The compact table rounds Jev's axes to 72, 88, 84, and 55. The unrounded axes from September 29 stay 72.0, 88.0, 83.8, and 54.7. JevK5 v0.3 is official rank 4 at 71.9 (A1). Plumb-4B is rank 5 at 71.6 (A1). Jev-Omni, still 71.5, is rank 6. decider-4b v2, still 71.3, is rank 7. Decision 4B v1.2 is rank 8 at 70.8 (A1). Imajev-4B is rank 9 at 70.4 (A2). Decision 4B v1.1 is rank 10 at 70.4 (A1). Manchego v2.1 is new at rank 11, 68.8 (A5). SemIf, still 68.7, is rank 12. spark-s1-4b-v6, still 68.2, is rank 13. Those addendum rows keep the addendum tag. The tie line is 69 of 77 adjacent pairs with a published comparison. 28 adjacent pairs have none. The capability control says 60 Jev-class systems. Method is still 904 open plus 720 sealed. We did not rerun it. The same URL on October 4 says v1.5.6 and 110 ranked systems of 116. Those three scores stay on top. imajev-4b is official rank 10 at 70.4, still tagged roster addendum A2.
Matilda, a self-run of Decision Index 0.2.1
59.26 · board Jev 57.91 · card median 56.6 ms
Maincode, September 30. The card calls Matilda Australia's first Jev-style model, Apache 2.0, 26.1B parameters, about 49 GiB in bf16. A self-run of the Decision Index 0.2.1 kit prints 59.26, raw 68.89, breadth 58.06, against Jev (hosted) at 57.91, 68.09, and 57.08. The card says the model is being submitted, so 59.26 is not a row in the September 28 board file. Knowledge and Reasoning is 43.52 against Jev at 51.40. The card counts 150,317 requests, all answered. The card's median is 56.6 ms. The post's text says 56.8 ms. The run section still says the weights are private. Opened October 8, 2026, the card prints Decision Index 0.3 at 60.16, and the model API lists private false. Dave Lemphers's "top Jev spot" is this index, not JevBench. We did not run the model.
Jeeves, public JevBench hard 0.865
hard 0.865 · Jev column 0.730 · transfer 0.746
PostHog/jeeves, MIT, Nicholas P. Waltz, 2026. Qwen3.5-9B LoRA, rank 16, plus a pointer head. Thinking, greedy, 2,560-token cap. Public JevBench hard, 111 items, sealed tier left out: Jeeves 0.865, the Kev-published Jev cell 0.730, Kev-8B 0.451. Test overall 0.889 against Jev 0.857. Transfer overall 0.746 against Jev 0.800. A separate 2,962-item split is 0.840 with thinking and 0.804 without. One H100: about 0.3 seconds without thinking, median 3.3 seconds with it. K.Chatani's post restates 0.730 and 0.865 and has no photo. We did not run it.
jgrep tests, 60 commits
recall 0.926 · selection 0.119 · $0.1112
kyu1204/jgrep, MIT, command jgrep, npm name jevgrep. The README says this is not dzhng/jevgrep, whose command is jg. Re-run September 28, jgrep 0.5.0, 60 commits: mean recall 0.926, code-only 0.449, selection 0.119, total $0.1112, mean 0.59 seconds. MINT's reply rounds those to 12%, 90%+, and $0.11. The model field is jev-latest. The CLI does not echo the response model id. zod's mean moved from 0.899 to 0.882 with no package hit, which the README calls variance between runs. We did not run the bench.
Ollaya, local server on port 11435
winnow:e4b 0.722 · Jev 0.738 · 89 ms
ollaya-dev/ollaya, Apache-2.0, September 30. The README says it is independent of Ollama and of TypeSafe. POST /v1/systemone on port 11435. winnow:e4b is 0.722 against TypeSafe Jev at 0.738, and 89 ms for five questions on an RTX 4090. The item count for that pair is not defined on the page we read. decider:4b is 0.680. The default 2B is 0.591. laya:en is 8 to 10 ms for five questions on an RTX 4090. fp32 exports match PyTorch on 100% of 2,383 questions per checkpoint. Winnow's JevBench 73.2 is a different number. We did not install it.
Jev as a judge, v3 cascade
1.7 points behind · cascade +0.9 on 1,610
Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman, arXiv:2609.26550v3, revised September 29. On 4,850 public judgments JEV is eighth of fifteen LLM judges, 1.7 points behind GPT-6, at $0.044 per 1,000 judgments, median 0.15 seconds. A frozen cascade is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of the fee, escalating 31%. Table 1: JEV 1.13 valid 5,172 of 5,172, GPT-6 Astra 5,171 of 5,172 at $12.182. JudgeBench paired difference is -14.6. A September 28 post quotes an RM-Bench hard/normal pair we did not find in the v3 sections we opened. The LangChain weather-agent repository is a separate project. We did not rerun it.
JevAdvBench, one appended opinion
12.1% flipped · command 10.1% · 812 questions
Jianyi Hu and coauthors, arXiv:2609.31142, submitted September 25. Isao Takaesu posted it September 28. The abstract: 812 typed questions, 66 scenarios, 9,744 single-edit variants, model jev-1.13.0. Rewording stays within 1.2 percentage points of the re-run baseline. Fields outside the schema never reach the model. One unverified opinion appended to the state flips 12.1% of decisions, statistically tied with the strongest injected command at 10.1%, and pushes 38% of confident answers below 0.8. The license line is the arXiv non-exclusive license, not CC BY. We read the abstract, not the 19 tables. This is a different paper from the 510-case InjecAgent attack.
OpenAI Decisions API, public beta
$0.10 / million input · output uncharged
OpenAI Developers, October 6, 20:47 UTC. The Decisions API is in public beta for all developers, powered by GPT-6 Luna. The September 29 post had said limited preview. The guide, opened October 6, prices input at $0.10 per million tokens and lists no output charge. Choice and score answers include confidence. Predicate answers return a probability. Images in the guide are inline base64. The New Stack extract still supplies the 150 millisecond and 1.6 second pair. The DevDay recap, opened October 6, still said limited preview. A request stays on OpenAI's API.
OpenAI Jev, October 6 call
HTTP 200 · 354 ms header · 420 input tokens
October 6, 2026. POST https://api.openai.com/v1/decisions with model gpt-6-luna returned HTTP 200. The choice was billing, confidence 1.0. The predicate probability was 0.98, with no confidence field. The score was 1.15, confidence 0.78. Usage was 420 input tokens and 0 output tokens. The header openai-processing-ms was 354. The clock on this desk was 1,858 ms. An empty object returned HTTP 400, missing model. GET /v1/models again listed 141 ids, including gpt-6-luna, and none contained decision. October 2 and October 4 calls had returned HTTP 403. The guide and the reference returned HTTP 200. openai-python 3.26.0 lists Decisions support. We did not install it, and we did not send an image.
Decisions and Jev, three texts
cat 0.98 and 0.98 · technical 0.99 and 0.74
October 6, 2026, one call each. The cat sentence scored 0.98 on Decisions and 0.98 on Jev. The TypeSafe quickstart's Stripe sentence scored urgency at 0.99 and 0.98. On three questions both chose technical, confidence 0.99 and 0.74. The leftover probability was 0.01 on sales for Decisions and 0.17 on billing for Jev. Frustration matched at 1.0, all of the weight on Frustrated but civil. The quickstart's printed response is a different illustration: confidence 0.78, technical 0.85, sales 0.0, billing 0.15, noul 1.0. Decisions processing headers were 407 ms, 773 ms, and 66 ms. Desk clocks were 1,641 ms, 1,596 ms, and 476 ms on Decisions, and 694 ms, 679 ms, and 649 ms on Jev. One sample is not a latency ranking.
Violoop, one QuickBooks bill
127.3 s · $0.0027 · Opus 978.9 s · $1.38
Violoop, September 29. Same harness, 123 decisions each, 30 of 30 rows. Jev 127.3 s, median 390 ms, estimated model cost $0.0027. Claude Opus 5.5 978.9 s, median 7,181 ms, $1.38. The page's 7.7 times is those two clocks. Tetris on the same seed is 282 pieces against 71, with Jev on a faster gravity setting, 0.2 s against 0.4 s. Comments: 450 in 30 s, about $0.014, no accuracy. A form: 30 of 30 in 97.6 s, about $0.002. The byline is BVIO marketing. We did not rerun the four tasks.
Polylane, five on-call steps
p90 4,752 ms to 508 ms · $0.762 to $0.464
Vi Tran and Boris Tane, Coreplane Labs, posted September 28. jev-1.13.0 on five fixed-answer steps. The rollup is p90 4,752 ms to 508 ms, 89.3%, and $0.76199 to $0.46369 per 1,000 calls, 39.1%. A later per-model chart does not match the task rows. No accuracy and no call count. We did not rerun the traffic.
JevScout, oversized MCP text
agent time -38% · agent cost -26% · comments 1/5 to 5/5
Tim Lin, yctimlin/JevScout, MIT, September 29. Claude Code hook. Default scope is an MCP result of at least 100 KB. The post says 7 tasks, 38% less agent time, 26% less agent cost, no correctness loss. The printed table has four rows, five runs each. One hundred comments at 196 KB go from 1 of 5 to 5 of 5. TypeSafe's own tokens, 9,000 to 81,000 on the smaller rows and about 300,000 on a 5.2 MB response, are billed separately. The evaluation note does not name the other three tasks. We did not run the hook.
DTap, Jev as a tool selector
direct ASR 70.1% to 36.1% · benign 60.9% to 55.8%
Zhaorun Chen, September 27. The chart does not print a task count. Direct misuse: Jev 1.13 tool selector 70.1, self-gated 36.1. Indirect injection: 43.5 to 21.6. Benign success: 60.9 to 55.8. Latency per call is 0.32 s for both Jev rows. The public leaderboard listed 0 agents when we opened it on September 29. This is a different chart from the 510-case InjecAgent paper. We did not rerun it.
Solar Decide, public 231
87.0% · p50 0.14 s · p95 15.7 s
Sung Kim, September 29. The chart says public JevBench 231, easy 48, standard 72, hard 111. Solar Decide 87.0% at p50 0.14 s. Jev 1.13 through OpenRouter 86.1% at 0.30 s. The footnote puts p95 at 15.7 s against Jev at 0.40 s. 87.0% of 231 is 200.97, so the percent is a rounding, not a count we saw. Open-Jev's published Jev cell on the same 231 is 200 of 231, 86.6%. AskUp is 20 of 20 against a keyword rule at 14 of 20, with no Jev cell. We did not rerun the 231.
Glean, four internal decisions
Recall@6 40.2% to 44.3% · 40 transfers, 8.1×
Tony Gentilcore, September 28. Rerank of up to 50 results on 4,855 queries: about $0.00044, p50 0.195 s, identical scores on about 37 of 41 candidates. Production order as a tie-break raises Recall@6 from 40.2% to 44.3%. Forty transfers where the old path actually handed off: median speedup 8.1 times. The routing accuracy on 751 entries is in a chart the copy did not include. Citation judging is one 1,448-word answer, three runs, and the article does not treat it as an accuracy result. We did not rerun the sets.
Pi skill picker, 20-skill demo
about 1,000 tokens off the first prompt
Arnav Gupta, championswimmer/pi-skill-picker-jev, MIT, 0.1.0, September 29. A Pi extension asks a decision model which skill names to load. The README's storefront demo cuts the initial system prompt by about 1,000 tokens. The post's 250-skill work repo, about 10,000 tokens and under a second, is not in the README. If the decision service is down, the extension does not load the full catalog. The scoring note, opened September 30, uses a five-level Score and a default threshold of 0.625. We did not run the demo.
imajev-4b, JevBench scores
v1.5.6 rank 10 at 70.4 · v1.4.2.2 at 67.37
Mohit Garg, mohit67890/imajev, Apache-2.0, September 28. Changelog v1.4.2.2 adds Imajev-4B at 67.37, rank 1, on the unchanged v1.4.2 scorer. Jev stays 63.29 on that board. The capability view we opened September 28 lists Imajev at 66.3 and official rank 1, Jev at 64.7 and official rank 4. On v1.5 the same model is a roster addendum, would-place sixth at 70.4, capability 70.8, and is not in the frozen ranked order. A September 26 note moves ImajevBench from 82.4% to 83.9% and public hard as shipped from 70.3% to 72.1%. Unknown abstentions go from 14 of 21 to 18 of 21. DecisionBench ECE goes from 0.024 to 0.069. The README's DecisionBench screenshot prints 79.65, third of 56, with jev-1.13 at 71.90. We did not load that Gradio board. The page opened October 4 is v1.5.6. imajev-4b is official rank 10 at 70.4, still tagged roster addendum A2. Image JevBench v0.1.5 lists 76.4 at rank 2 of 50. Opened October 8, 2026, Image JevBench v0.3.0 lists capability 68.0 at rank 1 and composite 63.7 at rank 2 of 52.
Plumb-4B, public hard and the official composite
89/111 · official 65.84
Christopher House, crh225/plumb, Apache-2.0. His runner on 231 public items, one RTX 4080 Super: v5 hard 0.802, 89 of 111, against a re-run of JevK5 v0.2 at 0.739. Eight fixed, one broken, McNemar p = 0.039. Held-out check 0.846. Hard ECE 0.094, worse than v4's 0.051. Changelog v1.4.2.1 scores the submission 65.84. The September 28 capability view lists official rank 2 on that board. On v1.5 the row is an addendum at 71.6, outside the ranked order. His table has no Jev column. We did not rerun it.
400M Kev, 17,476 rows, then a phishing set
79.96% · Jev 73.54% · phishing 50.25%
Sandipan Kundu, skundu42/kev. No LICENSE file in the root listing. Not Jared Palmer's Kev. Full paired run: 79.96% against Jev 73.54% on 17,476 of his rows, ECE 0.0151 against 0.0979, median 132.7 ms against 388.4 ms. A 100-row pilot went the other way, 80% and 81%. PhishNChips, Jev not called again: 50.25% against a published 62.6%, and 1,995 of 2,000 emails flagged. Typed decisions 44.15% against a published 72.7%. We did not rerun it.
jevgrep, ten SWE-bench tasks
8/10 both arms · total cost -25.8%
David, dzhng/jevgrep, MIT, package 0.4.3. Ten tuned Python tasks, one fresh attempt, gpt-5.6-sol medium. Official solves stay 8 of 10. Combined cost $5.6564 against a saved $7.6221, Jev's list-price estimate included, 25.8%. Sol alone in an older file is $5.44 against $7.62, 28.6%. The September 26 post said 40%. Five of the ten combined totals are higher than the saved baseline. We did not rerun it.
toolgate 0.15, write then execute
19/20 pairs · 1,369 decisions, 0 permissive misses
Risk Averse Technology Company, RiskAverseTech/toolgate, MIT. Set 6 on 0.15.0: 19 of 20, all 10 pairs separated, zero permissive allows. Set 7 is 20 of 24. Set 8 is 19 of 22. The 1,369-decision week is version 0.9.1, September 21 to 27, with 455 items labeled again blind. p50 on 1,368 direct calls is 1,060 ms. Defaults: deny 0.85, ask 0.55. We did not rerun the sets.
100 grant posts, one day's labels
75/78 nos hidden · 5/5 yeses shown
Deji, September 28. 100 posts, 7 of them a real yes on his first key. After full pages, an explicit doesn't-say, and one post at a time: 75 of 78 nos hidden, 5 of 5 yeses shown, 6 of 6 duplicate programs, deadlines on 33 of 41. About 2 cents. No repository. He changed at least one of his own labels. We did not see the posts.
jevbench.app, one recorded set
20/21 · miss at 0.51
Yusuke, jevbench.app, September 28. Not Florian's JevBench. Lead qualification: 21 examples, 10 yes and 11 no, recorded September 27. The index says Jev matched 20 of 21. The miss is a human no and a Jev yes at 0.51. Seven other tests have no match count on the index. Opening a test makes no new call. We did not export the rows.
Deel, eight quality bars
6 of 8 at parity or better · messy questions -16.6
TypeSafe, September 25, four charts attributed to Deel. No denominator and no named frontier model. Quality, in points: expense +36 against today's rule-based receipt matching, repeat questions +26.7, declining with no answer +8.3, well-formed metric pick +2, escalations at parity with fewer false alarms, PII at parity (100% both), 3-level tags -3.6, messy metric questions -16.6. The tweet says 50% to 86% versus human reviewers and 70% to 97%. Those four percentages are not on the chart. Cost against the frontier LLM in use today: 59, 49, about 45, and 20 times. The 45-times bar is estimated from per-token pricing. Speed: live-shadow escalation 4 times, metric pick 2.8 times, offline escalation 1.9 times, PII at parity. The tweet says offline tests were 2 to 3 times. We did not rerun the calls.
Decision hijacking, jev-1.13.0
validated ASR 1.8% to 3.5% · CHR at 0.7 is 0.2%
Tiantong Wu and Wei Yang Bryan Lim, NTU. arXiv:2609.28613v1, September 23, CC BY 4.0. 510 reconstructed InjecAgent cases. No GitHub URL in the text we read. DH-4 validated success, at least 3 target selections in 5 fresh calls, is 1.8% at budget 0 and 3.5% at budget 24. Intervals [0.0, 6.3] and [0.0, 10.0]. Mean best-so-far attacker probability goes from 0.043 to 0.084. Conditional hit rate at confidence 0.7 stays 0.2%. An enhanced override wording goes to 0% attack success. DH-3's largest registered gap, +0.0030, sits under the 0.02 effect they precommitted to. Claims are limited to jev-1.13.0. We did not rerun it.
RLCDAlignBench, generic Noul
AUROC 0.886 · ECE 0.168 · null 0.074
Ruoqi Guo, Yi Liu, and seven coauthors. arXiv:2609.29429v1, September 24, CC BY 4.0. sumleo/RLCDAlignBench, code MIT, data CC BY-NC 4.0, Hugging Face copy gated. 10 failure types, 44 benchmarks, 7,193 instances, all runs jev-1.13.0. Generic Noul median AUROC 0.886 [0.821, 0.952] on 31 benchmarks, 25 of 31 ahead of the better of length and TF-IDF. Median ECE 0.168 against a base-rate null of 0.074. One pass over 19 API-judge benchmarks is $0.30, 63 times less than those judges at list prices. AYi prints 0.31 seconds, $18.96, and 93% on the confident half. Those three are not in the tables we transcribed. We did not rerun it.
jevos-1b, 2,000 yes/no questions
0.815 · Jev 0.927 · Laya 0.489
Federico Elia, with Loris Salsi. feder-cr/jev, September 26. pyproject says MIT. The root listing has no LICENSE file. Q4_K_M GGUF, llama.cpp, POST /v1/systemone on 127.0.0.1:8017. Noul only. Choice and Score return 422. The printed bench is the README image alt: short latency 54 ms, 344 ms, 104 ms. Accuracy on 2,000 yes/no questions from unseen policies: 0.815, 0.927, 0.489. The question list was not in the files we opened. The base model is not named. We did not run it.
jev-code, three-task ladder
0 of 3 pass · hello world 3 turns / 16 requests
Roberto Montalti, rhighs/jev-code, September 26. package.json 0.2.0 has no license field. The root listing has no LICENSE file. Jev picks grammar productions. The guide's ladder, one run per task, dated September 18, commit 517d0f0: guessing-game fails at 38 turns and 2,000 requests, file-io at 34 turns and 3,000 requests, multi-file at 60 turns and 2,516 requests. Search width 3 fails the first two. Hello world passes in 3 turns and 16 requests. Bash is not sandboxed. The guide says the eval does not run in CI. We did not run it.
astra-jev-harness, one fixture
Astra input -27.0% · checks 6/6 · files 28 to 4 were 1.2%
Louis, Oranquelui/astra-jev-harness, MIT, version 0.5.0. September 24 completed run, one trial per arm, commit 9eb4229: Astra input 19,198 to 14,011 on Extra High, output 193 to 126, checks 6/6, API-equivalent USD 0.201630 to 0.146694928. Medium elapsed time rises from 9.074 s to 12.217 s. The September 22 note cut files from 28 to 4 and Astra input only from 42,714 to 42,199. Selection kept 75 bytes instead of 27,353 and raised Jev input from 392 to 6,784 tokens. The dollar column is his formula, not a Codex invoice. We did not rerun it.
Decision Index 0.2
0.2 Jev 51.67 · 0.2.1 Jev 57.91
apolinario/decision-index, MIT code, edition dated September 24. Forty benchmarks, five equal areas, chance-corrected. Unanswered counts as wrong. A tie is 0.25 points. The kit's test locks Jev at 51.67, AutoJev-27B at 50.94, Hopper at 30.01, and Verdict at 1.82. The README on September 30 says a rescore covers 64 entrants on the 0.2 board and 67 on the 0.2.1 board. September 25 posts put Drex at 51.73 and first on 23 of 40. That 51.73 is not in the test we read. Movez labels his flap-or-glide run simulated. Florian says Jev is first on raw intelligence here and on JevBench. Not the 63.29 harmonic mean. On September 30 we ran the kit tests: 13 passed in test_index02.py and 19 in test_index021.py. The board file generated September 28 is edition 0.2.1 and lists Jev 1.13.0 at 57.91, first of 71, with no row named Drex. The September 27 file still has Jev at 51.67, and Decider chat Gemma-4-31B at 51.93. Nace's page prints Drex 1.5 at 58.28. One POST through the kit's http engine to api.typesafe.ai/v1/systemone, model jev-1.13.0, state "The color is red.", came back status ok: choice red at 1.0, noul 0.94, 324 input tokens and 48 output tokens. We did not run the 120,340-request suite.
Jevrail, model off
22/22 catastrophic · 15/26 false asks
Saman Pandey, SamanPandey-in/jevrail, MIT, September 25. Hook in front of Bash. Sample config pins jev-1.13.0, ask at 0.35 and deny at 0.80 on destroys_uncommitted. The printed eval is --no-model on an 80-entry corpus: catastrophic recall 22/22, tier-0 hard denies 12/22, false asks on safe commands 15/26 (57.7%). The note says the cuts stay unproven under 300 entries. No model-on row. The 0.91 in the README is an example. Author-run. We did not run the eval.
Lightpanda, hostel memory
161 MB · Chrome 1,654 MB · Jev clock not in the file
Lightpanda, September 25. The post says 129 agents against 12 Chrome, 11 times less memory, and 6.7 seconds with Jev against 14.1 seconds with gemini-3.8-flash. The gist measures resident memory and does not call a model. Its note for the default Hostelworld page is 161 MB against 1,654 MB, marked 10 times, on an 8-core laptop with about 20 GB free. booking.com would print a fictional 36 times if both sides hit a challenge. We did not run the script.
CLM-8B, verifier on 38 DeepSWE holdouts
31/38 · Terminal-Bench 87.6% of 30
Jacky Kwok, Contrastive-LM/CLM, Apache 2.0, September 23. Frozen Qwen3-8B, two 20M heads. Zero-shot line: on par with Jev, up to 9 times lower latency. Thread: about 1,000 candidates is 13 times faster. Verifier: Opus 5 candidates on 38 held-out DeepSWE tasks, 31/38 = 81.6%. Fable 5 candidates on 30 Terminal-Bench 2.1 tasks, printed as 87.6% with no fraction. Jev below pass@1 on that setup. H100 latency 4.1 to 5.7 times Jev. The post said 4 to 6. RTX 4090 cache: revisited state 1.7 ms to 0.6 ms. The May 26 DeepSWE agent board is a different job. A September 26 page reuses a historical CLM column on 1,253 function-selection requests: 76.86%, 58.42%, and 47.49% on full JSON. That column is not a new CLM run. Author-run. We did not run bon_eval.py.
Eikos, author hard tier and a 0.90 gate
82.9 · Jev 73.0 · error at 0.90 is 2.4%
Caio Vicentino, caiovicentino/eikos, MIT, September 24. Same items in his harness. Public JevBench hard tier: Eikos-27B 82.9, Eikos-4B 72.1, Jev 73.0, Laya 35.1. No denominator on that row. Error at confidence 0.90 or higher, 7,140 items, 6 suites: 2.4%, 2.4%, 5.9%, 37.3%. Decision hidden in 64k tokens, mean of 3 positions: 88.3 and 74.2. Jev's cell is blank (README: API limit 32k). The post says eight batteries. The confidence row says six suites. 4B base is Qwen3.5-4B. 8-gram decontamination against public JevBench. Release omits 2% of training rows. vLLM before 0.30 dropped 3 to 6 points on long batches. Chess screenshot, 22:54 UTC the same day: 388 games, Eikos 42 wins, Jev 12, 334 draws, score 53.9%, Elo +27 (95% interval +14 to +40), answer time 3.5 s and 288 ms. Arena file at 09:20 UTC September 26, cycle 626 of 864: equity $9,653.19 and $10,051.22, Brier 0.2716 and 0.2500, hit rate 50.2% and 45.9%. The September 25 file, cycle 302, was $9,922.72 and $10,125.97. On September 30 the model card's RuleArena row is 0.50, 0.50, and a blank Jev cell. The repo's only JSON file lists eleven label ids. Author-run. We did not run the suite or replay a game.
mizchi, fit table
0 illegal in 489 · 6 to 27 wrong pages
mizchi, jev-playground docs/fit.md, September 24. No LICENSE file at the repo root. Enumerated moves: 489 MOBA decisions and 37 chess moves, 0 illegal. Choice then Score: 19/24 to 23/24. Names 90.0%, plus one line 100%. Unlisted defect classes: atomic questions 11/15, general question 15/15. One-line monitor rule 16/16 and 0 wrong pages; Jev 6 to 27. Named API question still no at 0.09 to 0.22. Adjacent enemy, when true, 49%. Five components 26 to 103 times faster, and the page says no quality column clearly wins. 1,000 functions $0.017 to $0.055. The linked criteria note, opened September 30, puts hard named bugs at 11/21 against held-out 3/15, 52.4% against 20.0%, about 2.6 times. Author-run. We did not re-run the cases.
JevBench v1.4.2.2, the board before v1.5
v1.4.2.2: Imajev 67.37 · Jev 63.29
Benchmark Heaven. v1.4.1, September 24, ranks 77 systems on 534 public decisions plus 308 sealed. Equal-weight harmonic mean. Jev 63.29, public 86.6%, sealed 36.7%, axes 53.1 / 76.3 / 83.3 / 52.0, $0.040 per 1,000, p50 0.65 s. v1.4.2, changelog September 25, kept those Jev cells and ranked decider-4b v2 at 64.13. v1.4.2.1 adds Plumb-4B at 65.84. v1.4.2.2 adds Imajev-4B at 67.37, rank 1 on that scorer. The capability view we opened September 28 lists Imajev at 66.3 and Jev at 64.7. v1.5.0, above, replaces this order. 63.29 is not the v1.5 score. The v1.2.8 geometric mean was 75.4. We did not rerun it.
42 claims, five runs
hard 73.3% · GPT-5.4 70.0% · p = 1.000
Jamie Watters, TheWayWithin/jev-bench, MIT. He calls this harness JevBench. The 534-task composite on this desk is a different project. Five runs, September 22, 18:29 to 18:48 EDT, 1,596 calls, $2.2510. Hard 18 means: Jev jev-latest 73.3% (72.2 to 77.8), GPT-5.4 70.0% (66.7 to 72.2), Sonnet 5 66.7%, Gemini 3.1 Pro 63.3%. Easy 24: frontier models 100% every run, Jev 91.7% every run. Majority-of-five McNemar against GPT-5.4 is 3 and 2, p = 1.000. Per claim: Jev $0.000025, GPT-5.4 $0.001289. Three Nouls in one call scored 77.6% against a single Choice at 83.8%, p = 0.219. Label flips: 2 of 42, under confidence 0.47. The September 22 post links the scratchpad note. The five-run table is the September 23 page. Single annotator. Author-run. We did not rerun the calls.
1,072 translation scores, one Jev pass
$0.3165 · 7,536,547 input tokens · 5m 23s
七誌, 7shi/multilingual-reader, CC0. jev-1.13.0, scheme degrees@f518286e, run dated September 22 and posted September 23. 16 models by 67 languages, 1,072 evaluations. Output tokens 5,561 on every model. Wall time 5m22.6s. $0.000295 an evaluation. The note attributes a 1.12-point run-to-run range to experiment 12, Pearson 0.998 for one run against the median of three. Port check, 268 translations: mean signed difference -0.006, 12 bit-identical. gemma4 core averages: German 89.88, Japanese 87.00, French 86.93, Chinese 86.20, Spanish 84.92. Finnish on the 16-model grid: Luna 91.6, gemma4 90.6, Bonsai 2 27B 3.8. Jev is the judge. Author-run. We did not rescore a language.
Local files, NFCorpus and SciFact
nDCG@10 0.322 to 0.377 · 0.687 to 0.745
andre, assadiandre/jev-search, September 23. No LICENSE file in the repo root. 623 queries, 66.3¢ total. NFCorpus 3,633 documents / 323 queries: nDCG@10 0.322 to 0.377, top-5 relevant 70.9%, top-100 relevant documents 25.5%, 0.084¢, 0.82 s. SciFact 5,183 / 300: 0.687 to 0.745, 80.3%, 92.9%, 0.130¢, 1.10 s. Only the nDCG row is printed as BM25, then JEV. JEV reads the top 128. A local desktop probe: 21,595 entries, 3.88 s, no API call. Dividing 66.3 cents by 623 is about $0.0011 a query. The post says under $0.001. Author-run. We did not run a search.
One needle, 100,000 lines
1,081 s · 1,480 s · same answer
El Asombroso Nosequien, elalber2000/jev-rlm, MIT, forked from alexzhang13/rlm-minimal. Seed 42, one generated file, deepseek/deepseek-v4-flash-0731, OpenRouter. Jev arm: typesafe/jev-1.13, 82 Jev calls, 1 recursive call, 1,081.05 seconds, answer 2867825. Other arm: 78 recursive calls, 1,480.50 seconds, same answer. The post estimates $0.155 and $0.089 off-peak. results.json has no charge field. The checked-in flag is use_jev = False. README names data/reslts.json. The file on the branch is data/results.json. One task. Author-run. We did not run it.
Open-Jev 27B v1.1, public 231
197/231 · hard 80/111 · Jev 200/231
Zefan Cai, model card ZefanCai/Open-Jev-27B-v1.1, September 23. Qwen3.8-27B LoRA plus a scalar head, 15,471,617 trained parameters, Apache-2.0 adapter. Public JevBench slice, 231 of 534, separate 16,384-token protocol: 197/231 (85.28%), hard 80/111 (72.07%). Jev 1.13.0 on that card: 200/231, hard 81/111. Released 2B and 9B stay 150/231 and 179/231. Internal hard-label audit, 127,787 rows: older panels 9,876/10,046 (98.31%) and 14,825/15,446 (95.98%); expanded 41,357/42,789 (96.65%) and 80,934/83,924 (96.44%). Public data omits 2,053 Wikispeedia rows. No latency row. New 2B training stopped. New 9B training has not started. Not the 534-task composite. Independent harness, author-run.
Classical sentences, four judges
+38.3 · +11.3 · +36.8 points
vase.park, hw725/dansa-research results note, September 23. jev-latest, one sentence per call, O at probability 0.5 or higher. 28,074 translated sentences: section sizes 5,212, 22,270, and 592. Jev target/control O rates 43.6/5.3, 34.4/23.0, 86.8/50.0. GPT-5-mini gaps +46.5, +3.8, +36.8. Gemini 2.5 Flash +39.3, +1.0, +22.0. Claude Sonnet 4.6 +47.0, +18.5, +31.8. Book-stratified odds ratios with Jev added: 8.64, 2.17, 1.78, intervals above 1. 28,089 calls, about $0.54. 342 items sit on 0.5. No gold label. No LICENSE file in the repo root. Solar Mini 4 on section 3 only: gap 4.1 points, 576 of 592 at 0.95. Author-run. We did not rerun the calls.
100 English items, Laya and Jev
Laya 92/100 · Jev 100/100
码农暖爸, September 23 post. 100 English judgment items. Laya's English checkpoint 92 correct and 8 wrong. Jev 100 correct. The post names feature descriptions, negation, mention versus intent, and conversational padding, with one example each. The author says this is not a full benchmark. The item list is a reply image, which this desk did not transcribe. No repository. An earlier post the same day prints no count and says a local Laya clock is not the same measurement as a cloud Jev call. Author-run.
Jevsus, true/false diode
209,628 calls · $3.13 · 45% range_wide
2nd1st, 2nd1st/Jevsus, MIT, jev-1.13.0, run dated September 20 and posted September 23. The post counts 3,608 statements, 40 forced picks, 521 names, 190 sentence templates, 209,628 calls, $3.13. The README says about 210,000 calls, 0 errors, and 8/8 anchors at 0.00 and 1.00. 45% of statements move by 0.20 or more across three wordings. Xi Jinping on "is a dictator" is 0.01 on that sentence and 0.50 across eight ways of asking. Stance sensitivity on "The United States is a racist country" is 0.75. 187 of 335 names under sixteen accusation sentences are not quoted. The README says a stance value is the model's output under that prefix. Independent harness, author-run. We did not rerun the calls.
Rizzo Flow, SemIf fixtures
0.812 · SemIf 0.819 · 49 ms · 6/36
Simone Rizzo, Rizzo-AI-Academy/rizzo-flow, Apache-2.0. Spark-X2.5-4B Q8_0 on llama.cpp, Windows 10, RTX 5060 Ti. authored144 balanced accuracy 0.812, held-out half 0.793, against SemIf's published 0.819 and 0.811. Paired difference -0.007, interval -0.076 to +0.065. perturbations108 is 0.848 against SemIf 0.766. Missing evidence: 6 of 36 confident and wrong, SemIf's published count is 1. Short-decision p50 49 ms, p95 52 ms, 5.6 GiB. At most 26 options. Uncalibrated by default. The 1.7B row is 0.678. Mac and Linux are untested on this runtime. The September 23 post is @QingQ77. Independent harness, author-run.
Ámbar Reflex, three local models
24.9% at 408 ms · 9B accuracy 82.7%
Abel Coronado, abxda/ambar-reflex-sistema-1, MIT, RTX 3060. Typed path, N=5: mean progress 24.9% at 408 ms. Same-size model writing JSON, N=3: 10.3% at 1,041 ms. Random, N=5: 4.6%. Offline, 554 labelled decisions: Qwen3.5-4B Q8_0 77.8%, Gemma 4 E4B Q6_K 81.6%, Qwen3.5-9B Q5_K_M 82.7%. Realtime progress for those three: 24.9%, 14.4%, 17.5%. Gemma jumps on 74% of decisions. Questions were tuned on the 4B. No hosted Jev column. Author-run. We did not play the game.
jev_local, JGLUE test
Sarashina Q4 34% / 86% · LFM 17% / 69%
あるごす, Argos1111/jev_local. JGLUE test on one API: LFM JNLI 17% and JCommonsenseQA 69%; Sarashina2.2 Vision 3B Q4 34% and 86%; ModernBERT-Ja, trained on the train split, 93% and 92%. The README says this is not a fair ranking. SARASHINA.md says JNLI is under a majority baseline. Choice cap 26. The post times the 3B model at about 100 ms when fast and about 500 ms typically, on a Radeon AI Pro R9700. The README prints 2.5 to 4 times LFM and does not print that pair. No LICENSE file. No Jev column. Author-run.
OCR computer use, one screenshot
$0.0002 · Opus 5 $0.032 · 155x
awlevin/typesafe-computer-use, MIT. Posted by Somi. One decision, same screenshot and goal. Jev: 4,882 input tokens, $0.0002, 0.13 to 0.38 s, about 1.5 s end to end. Claude Opus 5 on the bare image: 4,785 tokens, $0.032, 5.2 s, about 5.5 s. With history, Opus is $0.035 to $0.08. A 12-step task is $0.003 against $0.40 to $0.90. The page says Opus read dates off the pixels and this loop annotates them in code. OCR matched 7/13, 7/88, and 3/85 DOM labels. No task success rate. Windows adapter untested. Author-reported prices. We did not run it.
Jevre, 12,000 rows
test 0.9323 · held-out 0.8366 · ECE 0.0944
Di Zhang, trotsky1997/jevre, MIT, 0.85B checkpoint. 12,000 evaluation rows. Test accuracy 0.9323, ECE 0.0079, Spearman 0.976. Held-out sources 0.8366, ECE 0.0944, Spearman 0.976. State shuffle: 0.4282 and 0.4369. Served path 0.9267, 77 ms p50, 109 ms p99. Training run: one epoch, eight GPUs, 47 minutes. The post says RLCD and a parallel sampler. The README names Plackett-Luce ranking and a Brier term. No Jev column. Author-run.
rev, public JevBench tiers
hard 0.784 · Jev 0.730 · 2B hard 0.550
Steven Yu, 54yyyu/rev README, MIT. Public tiers, counts 48 / 72 / 111. Qwen3.8-27B INT4 on 2x L40S, exact logprobs, 2026-09-22: easy 1.000, standard 0.986, hard 0.784. Thirty-two samples match that row. Qwen3.5-2B, 8-bit, M2 Pro: 1.000 / 0.764 / 0.550. Jev 1.13.0 on the same table: 1.000 / 0.986 / 0.730, and the hard cell also prints 0.721 measured that day. temporal_numeric 0.400 against Jev 0.267. A 0.9 gate on the 27B note covers 78% of standard items and 35% of hard items, both at 1.000 on the answered slice. The README says this is not the public leaderboard. Speed for the 27B path includes two network hops: short p50 145 ms against Jev 354 ms, hard 317 ms against 326 ms. Local 2B hard p50 821 ms against Jev 326 ms. Clipboard 160/160. Calendar 216 items at 0.986. Mail AUC 0.941 on 146 messages. Independent harness, author-run.
hotdog-27B, two binary sets
1,603/1,631 · Jev 1,587/1,631
Michael, opensporks/hotdog-27B card, September 22. Rank-16 LoRA on dense Qwen3.8-27B, A/B logit read, temperature 1.0. Independent holdout, 1,631 decisions in 1,002 groups: 1,603/1,631 (98.28%) against Jev 1.13.0 at 1,587/1,631 (97.30%). Paired difference +0.98 points, 95% interval +0.19 to +1.79. Brier 0.013175 against 0.026884. ECE 0.58% against 6.40%. Public native-binary JevBench, 74 decisions in 62 groups: 60/74 (81.08%) against 62/74 (83.78%), -2.70 points, interval -12.20 to +5.63. Brier 0.125090 against 0.106305. ECE 11.95% against 6.54%. The card says the public row is not a JevBench win. The holdout shares source datasets with training. Latency is not compared. Development set 1,468 decisions. Selected step 96. Independent harness, author-run.
spark-s1-4b-v6, public-tier proxy
83.1 · hard 0.595 · Kev dev 0.756
Abhishek Rai, abhishek085/open-spark-jev, Apache-2.0. Qwen3.5-4B LoRA. Public items, easy 48 / standard 72 / hard 111: bf16 1.000 / 1.000 / 0.595, NVFP4 1.000 / 0.972 / 0.622. Public-proxy Intelligence 83.1 and 83.2. The README says the judge tier is private and this is not the official composite. Changelog: Intelligence 74.2 to 83.1 versus v5-4b. The post's +13.8% hard-tier line is not a v5 cell on that changelog. vLLM p50 74.9 ms and 53.3 ms. Kev transfer-v4, 764 items: locked test 0.787 against published Kev-4B 0.806. Development partition 0.756 against published Jev 0.857. Sixty tool calls: 0.900 against a recorded Jev 0.917, p50 74.9 ms against 421.6 ms. Own held-out accuracy 0.929 on 11,792 training rows. Independent harness, author-run. The Jev cells on Kev's suite are Kev's published figures.
Three Japanese sets, one posted table
96.0% · 90.2% · JamC-QA 56.0%
t.toda, September 22 image. Jev: JCommonsenseQA 96.0%, JMMLU 90.2%, JamC-QA 56.0%, mean 80.8%. Bonsai 2, one-token logit: 93.8% / 66.8% / 41.2%, mean 67.2%. GPT-5.6 Luna: 97.2% / 82.5% / 76.5%, mean 85.4%. The mean is the average of the three printed percentages. No item counts, no prompt, no repository. Independent harness, author-run. We read the image.
25 probability phrases, 99 people
rank agreement 0.99 · gap 3 points
Mike Taylor, September 22 chart. Human ridges: D. Hails, The CIA was Probably Right (2026), n=99, 25 phrases. Jev 1.13.0, one Choice per phrase, 0 to 100 in steps of 5. Caption: rank agreement 0.99, median gap 3 points, about the same as between two human surveys. Both series smoothed with a Gaussian kernel. No per-phrase mean is printed in the caption. Sasha Sheng replied that the chart is pretty. We read the image. On September 30 we opened the Hails essay. Probable is 70 there, rounded to the nearest 5, beside the same 70 in the NATO 1970s column and in Zonination 2015. The essay has no Jev column. We did not rerun the 25 questions. Author-run.
ASO Workbench, 578 held-out terms
Spearman 0.74 · mean error 10.4
Mustafa Kendigüzel, docs/POPULARITY.md, model of 19 September 2026, 44,658 training rows. Terms with autocomplete clues, 578 held out: Spearman 0.74, mean error 10.4 points, 81% within one band. Random 20%: Spearman 0.45, mean error 15.6. Turkey held out: 0.37 and 17.2 points. A held-out topic: Spearman 0.27 to 0.35, mean error 20 to 30. Jev through OpenRouter supplies brand, demand, and language-fit features. Without that key the app does not guess. About 55,800 labelled terms from five countries. The score is agreement with Apple's popularity, not a label from Jev. Author-run.
djev-run, Cloud Run latency note
35-60 ms · 100-123 req/s · $3.19/hr
Daniel Lee's taeold/djev-run README, amplified by @googlegemma on September 22. RTX PRO 6000, 20 vCPU, 80 GiB, min instances 0. Single request about 35 to 60 ms at samples=1, about 160 ms at samples=auto. Batch about 100 to 123 requests a second at concurrency 32. Cold start about 47.5 seconds. Price on the page $3.19 an hour while active. The post says roughly $3. POST /v1/systemone. No accuracy table. The repository lists no license. Author-reported timings. We did not deploy it.
Zachi's pg-jev, 2,000 rows and a batch check
3.5 s · $0.012 · 100% at batch 20
Zachi, realZachi/pg-jev README, PostgreSQL License, version 0.2.0 dated September 18. A 2,000-row table from Europe, about 190 ms to the API: first run about 3.5 seconds, 100 requests, about 296,000 input tokens, about $0.012. Cache about 50 ms. LIMIT 3 on a new condition about 0.6 seconds. Version 0.1.0 was 8.5 seconds and 338,000 tokens. Ground truth, 400 rows each, job title, EU membership, and a phrase in free text: batches of 1 to 20 were 100% correct, 40 were 92% to 98%, 80 were 77% to 94%. Default batch size is 20. pgjev.com still prints 40 per request and six concurrent requests. The September 17 post is a smaller run, 129 rows in about a second for $0.0009. Author-reported. We did not install the extension. TypeSafe's customer agreement limits published benchmarks of the Services. These figures are reported as published.
postgres-search, SQL rates, Jev unmeasured
82% first · top 10 88% · threshold 0.30
Alex, alexforman1/postgres-search README, MIT. Fifty handwritten queries on 440,302 USDA branded foods: a correct product is first 82% of the time and in the top 10 for 88%. Warm SQL: cheerios about 2 ms, milk 22 ms, cheerois 53 ms. The Jev step is one Noul per candidate in the top 10, default threshold 0.30, sink below the line, never sort. docs/jev.md says that pass was not measured because no API key was available. Fail open at 1.5 seconds. The 82% figure is the SQL order. Author-run SQL timings. The Jev columns are empty.
Decider 1, typed-decisions
0.768 · Jev 0.727 · sentiment 0.943/0.970
meraGPT blog, September 22, no byline. Post is @latent_node. Model sd-1. The blog says 400 cases, four workflows, five questions each, 2,000 decisions, zero-shot, one run. Jev 1.13.0 through TypeSafe's API. Accuracy / KL / Brier: Decider 1 0.768 / 0.096 / 0.052, Jev 0.727 / 1.442 / 0.148, ModernBERT-base fitted 0.646 / 0.223 / 0.119, MiniLM-L6 fitted 0.587 / 0.262 / 0.143, prior 0.470 / 0.347 / 0.189. By type, noul 0.840 vs 0.775, choice 0.733 vs 0.720, score 0.739 vs 0.696. Labels are the average of three teacher samples. Two samples agree 73.5%. Hugging Face viewer for LocalLLaMA/typed-decisions shows 400 rows in each of four subsets and 1,600 combined. The dataset card, read September 30, says the scored table is the test split, 400 cases and 2,000 decisions. The viewer's 1,600 rows are train and test together. We did not recompute the table. Median about 0.5 s. Cap 4,096 tokens, at most ten choice labels. $0.03 per million input tokens against Jev at $0.042. September 23 posts name a harness jev-eval, 300 cases per task, within 1 to 3 points, 2.7 to 4.3 times cheaper, about 0.7 s against 0.2 s. One printed task: yes/no sentiment, Decider 1 0.943 at $0.0078 per 1,000 calls, Jev 0.970 at $0.021. The thread does not link a results file. That name is not onlyoneaman/jev-eval. Vendor-reported.
dejevu, Zurich to London
5.63 s · published Jev 7.09 s · $0.0092
Idov Mamane, idovmamane/dejevu README, measured 2026-09-22, MacBook Pro M3 Pro, headless Chrome 153, OpenRouter. Median of verified runs. llama-3.3-70b on Groq: 3 of 3, 5.63 s, 10 calls, 9 actions, 15,071 input tokens, $0.0092. Published jev-ultrafast cell, Jev 1.13 plus Mercury 2.5, not rerun here: 3 of 3, 7.09 s, 17 calls, 84,650 tokens, about $0.0036. DeepInfra, same Llama, 2 of 2 at 37.5 s and $0.0020. Five other models verified 0 flights runs. Wikipedia: Groq 3 of 3 at 1.31 s; published Jev cell 2.80 s with the verified count blank. Race overlay 7073 ms vs 6070 ms. GIF caption 6.07 s. Search date moved to Sunday October 18 2026. TypeSafe backend in the repo is untested. Three runs per cell. Independent harness, author-run. The Jev column is the other repo's published figure.
Banking feed, 612 private rows
36.1% · 6 s · $0.023
Peter, @LordMarket22, September 22 post. 612 sanitized rows from a harder subset of about 33,000. Jev 1.13: 36.1%, 6 s, $0.023. Gemini 2.5 Flash: 37.3%, 95 s, about $0.088. Gemini 3.7 Flash: 42.5%, 78 s, about $0.180. Gemini dollars are estimates from the same text-token budget at public rates, excluding thinking tokens. No category list, no repository, no row file. Independent harness, author-run. The rows are private.
poker-table, 46 handwritten spots
p50 ~1 s · live pill 545 ms
Jinzhengxu/poker-table README. Spot bank: 46 spots, three repeats, typesafe/jev-1.13 through OpenRouter. Range reads described as matching a reasoning model, 6 to 40 times faster. Median about one second, against 6 seconds single-shot and 35 seconds for the tool-loop agent. Three passes under a cent. No accuracy percentage printed. The September 22 post's live pill is a different clock: 545 ms average and $0.002 a hand. On September 30 the live table was waiting for a hand and the Jev panel was not configured. The page does not contain 545 or 0.002. We did not play a hand or run the spot bank. The bb/100 table in the same README is a rule-policy ablation, not this Jev run. Independent harness, author-run.
AgentOS Pilot Router, 121 labeled turns
0.901 · local 0.355 · ~$0.005
AgentOS, September 22 chart in the announcement thread. Corpus tests/data/router_eval/cases.jsonl. 121 turns, Vietnamese, Chinese, and English, gold labels R0 to R3, classifier only, two repeat runs identical. Jev accuracy 0.901, under-routing 0.066, over-routing 0.033, macro-F1 0.894, R3 recall 0.907, ECE 0.062. Local MiniLM 0.355 / 0.413 / 0.231, and the chart leaves macro-F1, R3 recall, and ECE blank. English 0.946 against 0.513. Vietnamese 0.938 against 0.312. Chinese 0.806 against 0.250. One call per turn, about 0.9 s, 121 cases about $0.005. Opt-in. Turn text goes to typesafe.ai. A same-day follow-up, pull 3317, says it sharpens the R2 and R3 criteria. The chart is the announcement image. On September 30 that pull's file list has no results JSON. Its changelog prints accuracy from 0.760 to 0.901, and R3 recall from 0.56 to 0.91. Independent harness, author-run.
laya-mlx, one short question on an M3 Max
13.42 ms · 7.39 ms · 63/63
mizorewww/laya-mlx README. M3 Max, 40 GPU cores, 128 GiB, macOS 27.2, MLX 0.32.2, FP16, model load excluded. Laya 421M P50 13.42 ms, P95 13.92 ms, 146.8 questions per second at batch 64, peak 943.6 MiB. Multilingual 322M P50 7.39 ms, P95 7.79 ms, 395.0 questions per second, peak 687.6 MiB. Selected answers matched upstream Laya on 63 of 63 validation questions in FP32 and FP16. Snake at --optimize --max-speed: 75.40 moves per second across 2,400 moves, zero deaths, two safety corrections. The tweet's 60 decisions per second is not that line. No new table against hosted Jev. Independent port, author-run.
Spring AI TypeSafe, one laptop ticket call
310 ms · cookbook 111 ms
Christian Tzolov, Spring blog, September 21. Three-question support ticket, median 310 ms from his laptop. One question, median 275 ms, 73 output tokens on the three-question call. Blog sample: urgent 0.95, billing confidence 0.82, frustration Score 1.1. The README comments on the same ticket text show different values. He cites TypeSafe's self-consistency cookbook: 14 questions, $0.000043, 111 ms, against Haiku 4.5 at $0.0018 and 1.8 s. The blog says that comparison is TypeSafe's and still needs checking. Judge log on a scripted -455 C answer: plausibility 0.02, groundedness 0.89, helpfulness inconclusive at confidence 0.44. Author-run timings. The cookbook row is vendor-reported.
jev-rubiks, forty move picks
distance 22 to 22
maxli, maxlibin/jev-rubiks README. Jev asked for the best of 18 face turns, forty times. Kociemba distance 22 to 22. A random walk in the same note went 22 to 21. Kociemba in code is about 20 ms and at most 22 moves. The experiment was removed. The coach that shipped asks whether to speak. One screenshot caption: speak 0.83, warn_broke_progress 0.94. Nine live coach situations are described and not printed as a table. Independent harness, author-run.
MotherDuck prompt_jev, 100k AG News train rows
89% · 2,484 rows/s · $0.50
MotherDuck, blog dated 2026/09/21, no byline on the page we read. 100,000 articles, reservoir sample seed 43, from the AG News training parquet. Jev 2,484 rows/s, 89% accuracy, $0.50, 40s. gpt-5.6-terra 52 rows/s, 88%, $37.58, 31m 59s. gpt-4o-mini 84 / 80% / $1.93 / 19m 45s. gpt-5-nano 94 / 83% / $1.58 / 17m 49s. gpt-5.6-luna 61 / 84% / $3.53 / 27m 25s. Null choices are not scored as wrong. The null count is not printed. Training split, not a held-out test. Runs at 1 million and 10 million rows are described as similar, with no table. Vendor-reported.
jev-gateway, 120 chess-agent sessions
-57% output · Opus feature +83% time
Vini Lana. vinilana/jev-gateway-bench, cited from the gateway README. 120 sessions, six models, two chess-engine tasks, five runs with routing and five without, no MCP and no plugins. Medians versus the same model with routing off, output / input / time. Bug-fix: Astra -57% / -7% / -39%, Sol -57% / -40% / -36%, Luna -12% / -10% / +10%, Fable 5.1 -13% / -19% / +6%, Opus 5 -7% / -22% / +2%, Sonnet 5 -41% / -48% / -25%. Feature: Astra 0% / +2% / +8%, Sol -9% / -39% / -16%, Luna -14% / -51% / -14% (3 of 5 solved, against 5 of 5 off), Fable 5.1 -24% / -27% / -26%, Opus 5 +22% / +61% / +83%, Sonnet 5 +9% / +16% / +37%. Claude Code rows are hint mode. Five runs per cell. The README says the bench repo records one run caught copying from another. Independent harness, author-run.
llamacpp-jev, four questions on one image
526 ms · 32/32 · L4 236 ms
Chirag, NakliTechie/llamacpp-jev. Qwen3.5-2B-Q8_0, unmodified llama-server, 2026-09-22, M4 Pro 24 GB, Metal, GPU idle, four slots, cache-ram 0. Fresh 448 by 448 image, four typed questions, eight images: median 526 ms (525 to 546), 32/32. Same image 254 ms. Four text questions 238 ms. Idle NVIDIA L4, same commit: fresh image 236 ms (174 to 237), 32/32. September 21 on the same Mac with a busy GPU was 946 ms. Six hand-labelled photos, 22/22 scored, two unscored. 64-way questions on 0.8B and 2B both answered option 16. Probabilities are a raw label softmax. Not a JevBench row. Independent harness, author-run.
kishida/llama.cpp branch jev, 1,191 questions
Flash-Next 0.944 · MoE 92 ms
きしだ, kishida/llama.cpp docs/jev.md, opened October 3. Branch jev builds POST /v1/systemone into llama-server. 1,191 multiple-choice questions, 2 to 8 options, guessing 0.283. Temperature fit on 358 other questions. Most rows are an RTX 4060 Ti, -ngl 99 -c 16384 -np 8. Mac rows are another machine, and the page says not to compare their times with the 4060 Ti. Time is the mean of 50 one-question requests. Qwen3.8 Flash-Next UD-Q4_K_XL is 0.944, ECE 0.023 to 0.015, T 1.26, 292 ms, Mac. Qwen3.6 35B A3B is 0.915 at 101 ms. Gemma 4 26B A4B is 0.913 at 92 ms. GLM-5.3-Flash is 0.917 at 692 ms with an assistant prefix and 0.541 without it. Qwen3.5 2B Q8_0 is 0.709 at 52 ms. The October 2 screenshot omits Flash-Next. Opened again October 6. The quick start still says the endpoint is not in upstream llama.cpp. The newest commit on docs/jev.md is still October 1. Not a JevBench row. Author-run.
llama.cpp pull 29818, five decision GGUFs
3 ms · 5 ms · 12 ms · 36 ms · 43 ms
Xuan-Son Nguyen and Victor Mustar, Hugging Face note published October 2. Pull 29818 merged the same day. Median time for one question on one NVIDIA RTX PRO 6000. Julia-1 144M, 3 ms, Apache 2.0. Laya 421M, 5 ms, Apache 2.0. Kev-4B, 12 ms, Apache 2.0. lev, 36 ms, Apache 2.0. OpenJev 27B, 43 ms, images, CC BY-NC 4.0. Score takes 2 to 10 levels. The printed sample rounds billing to 0.9049, confidence 0.8574, noul 0.8208, score 2.2821, output tokens 0. Julia-1 routed a bare-label charge complaint to shipping, and to billing at 0.99 once the options had descriptions. Not the 448-image timing. Vendor-reported.
llama.cpp v0.6.0, Clef and Nimble
Clef images · v3 56.88
ggml-org/llama.cpp release v0.6.0, published October 5, 2026, 16:56 UTC, nightly b11429. The overview lists /v1/systemone for laya, julia-1, lev, openjev, kev, and nimble, and Clef for text and images. Pull 29844 merged October 2 for Bespoke-Nimble-9B-v3. That card prints Decision Index 0.2.1 at 56.88, CC BY-NC 4.0, a LoRA on Qwen3.5-9B. It is not the 292 of 324 holdout. Pull 29831 merged October 3 for ggml-org/Clef-GGUF and Clef-Flash-GGUF, and said images were still missing. Pull 29969 merged October 5 and adds images for the clef type. The server README at the tag names OpenJev and Clef as the image examples, caps a choice at 52 options for OpenJev and 255 for Laya and Clef, and says Clef answers every question in one prompt. A Clef server does not generate text. The October 6 Hugging Face table adds Clef at 27B, Apache 2.0, images, with no speed. Nimble is not on that table. Not the 3 ms to 43 ms row. We did not run the server.
Open-Jev, 231 public JevBench tasks
9B 179/231 · 27B 197/231 · Jev 200
Zefan Cai. Audited JSON generated 2026-09-21T06:27:47Z. Public slice of JevBench: 231 of 534 tasks (original 72, easy 48, hard 111). Released Open-Jev 9B 179/231 (77.49%), hard 66/111. 2B 150/231, hard 46/111. Jev 1.13.0 200/231 (86.58%), hard 81/111, one probability vector renormalized. GPT-5.6 Luna 206/231, hard 89/111. GPT-6 Astra 231/231, hard 111/111. P50 189 ms / 291 ms / 954 ms / 2,206 ms for 9B, Jev, Luna, Astra. Open-Jev on one H100, loopback, cache off. Hosted calls are HTTPS. GPT probabilities are verbalized. Candidate order differs on 119 of 139 Choice tasks. Temporal/numeric family: Jev 4/15, Luna 5/15, Astra 15/15. Derived estimates about $0.038 / $0.179 / $8.82 per 1,000 decisions. The September 23 card adds 27B v1.1 at 197/231 and hard 80/111 on this slice, with no latency. Not the 534-task composite. Independent harness, author-run.
jsort, CommonLit and 95 Fed statements
r 0.824 · $0.046 · rho +0.46
Khaled Eltokhy. CommonLit, 300 excerpts, Jev 1.13 through OpenRouter, 2026-09-19, uncached. jsort -k 10 Pearson r 0.824, Spearman 0.841, 1,500 calls, $0.046. One Noul per text r 0.754. SMOG 0.661. Teacher-scale ceiling near 0.88. Fed opening statements, 95 files, more hawkish about inflation: rank correlation +0.46 with the same-day funds-rate move, +0.37 with the next 180 days. Joe Weisenthal's September 22 post says he rescored 4,005 speeches. That count is not the 95-file script. Independent harness, author-run.
Security triage, 100 synthetic cases
65.3% · 75.1% · 91.2%
RZ, recorded run 2026-09-19-public-v5. One hundred synthetic TypeScript cases, five passes, 1,500 decisions, zero errors. Jev 1.13.0, gpt-5.6-terra in Codex, claude-opus-4-6 in Claude Code. Balanced accuracy 65.3% / 75.1% / 91.2%. Model-request p50 0.57 s / 10.23 s / 13.83 s. Same answer on all five passes: 95 / 76 / 91 of 100 cases. Jev class recall: insufficient context 93.9%, safe 45.5%, vulnerable 56.5%. Cost for 500 decisions $0.015 / $15.02 / $13.94. Labels by one author. Gap under about fifteen points is noise. Independent harness, author-run.
Jev RAG, shortlist gates
CLERC 5% to 18% · LoCoMo 0.950
TypeSafe re-ranking cookbook, jev-1.12. CLERC, 3,565 passages, 40 queries, BM25 shortlist of 30 already holds the gold on every query. Top-1 moves from 5% to 18%. Top-5 15% to 35%. Top-10 38% to 62%. 1,200 calls, $0.0645. Classifying cookbook, same model, August 27: 81 passages, top 12, four Nouls, one request per passage. The planted injection scores 0.99 and is excluded. Hindsight listwise Choice, LoCoMo, 30 candidates: recall@1 0.950 against MiniLM 0.800, 0.027 s against 0.12 s. Pruning is off and drops 19% of gold. aifabrice NFCorpus best row nDCG@10 0.451. The four-question gate there scored below ordinary reranking. Lyzr, jev-1.13.0, one call on 20 chunks: SciFact NDCG@10 0.836 against retrieval 0.776. Sufficiency AUROC 0.90 on SQuAD v2 and on HotpotQA. jevrag sufficiency, HotpotQA n=700, matches three fixed rounds and uses 38.5% fewer retrievals. ECE on that question is 0.3322. We did not rerun them.
jevsearch, TypeSafe docs ranking
83% Hit@1 · 41% keyword
Kyle McLaren. 109 TypeSafe documentation pages split into 443 documents, 41 labelled queries. jev-search 83% Hit@1, 83% Hit@3, 0.83 MRR@10, median 278 ms uncached. Keyword pass 41% / 59% / 0.51, median 8.7 ms, tied with Lunr. Intent Hit@3 73% versus 42%. 6,169 input tokens per uncached query, about $0.26 per thousand searches. Jev re-ranks a 20-hit pool; it does not retrieve. Independent harness, author-run.
duckdb-jev, live Choice throughput
2,049 rows · 0.887 s
Prasanth J. Live jev-1.13.0 Choice on DuckDB 1.5.5, macOS arm64. 1,000 rows at batch 100 and concurrency 10: median 0.515 s, 1,943 rows/s. Separate capture: 2,049 distinct serialized rows in 0.887 s (2,311 rows/s), cached replay 28.6 ms with zero API calls. Twelve support-ticket templates with unique IDs. Throughput and cache reuse, not general accuracy. Independent harness, author-run.
DocJev, 40 PDFs and eight packets
40/40 · 7/8 · 5.73× / 6.45×
Jerry Liu, real-small-v1-run01. Forty authentic public-sector PDFs, eight constructed packets, 116 pages, same LiteParse text for both engines. Classification 40/40 for Jev 1.13.0 and GPT-5.6 Luna; medians 138.6 ms versus 794.3 ms. Splitting 7/8 exact packets versus Luna 8/8; medians 209.6 ms versus 1,352.3 ms. Both found all 32 true boundaries. Jev added one extra cut on a Federal Reserve attachment. Estimated decision cost $0.011663 versus $0.046894. Human labels unreviewed. Small convenience sample. Independent harness, author-run.
DiffusionGemma as Jev, hand chart
198/201 · 196/231
Matt Mastracci, September 17 chart, eight hand sets. The printed fractions add to 198 of 201 for DiffusionGemma and 191 of 201 for Jev. One read on a warm DGX Spark is faster on 7 of 8 sets, after a 46 ms round trip is subtracted from Jev. Auto re-reads are slower on all 8. Hangzhi, September 21, one A100: 196/231 on the public JevBench tasks against the published artifact of 200/231. Text ECE 0.1222. ywchiu, 100 decision points times 5: djev-spark four-field joint 32.2% against Jev 61.4%. Chirag, sieve: reversing options flipped 24%. Josh Brown's chart: average precision 46.8% against 46.2% on 2,220 pairs. The Perch URL returned 404. Author charts and READMEs. We did not serve the model.
djev, JevBench hosted row
74.3 · hard 69.5%
Maisa hosted API on DiffusionGemma. JevBench v1.2.8: 74.3, third, Intelligence 88.4, Calibration 65.4, Speed 91.4, Cost 57.6. Hard-tier 69.5% against Jev 74.1%. p50 0.24 s on the production API. Announced $0.035 per million input, $0.026 per 1,000 decisions; nothing charged yet. Probabilities labelled experimental. Independent harness, author-run.
SimpleJev Qwen3.8-27B, JevBench demo row
67.3 · hard 75.0%
Featherless public demo, logit read, no generation. v1.2.8 rank 13 (Florian's v1.2.6 post had it at rank 9). Intelligence 89.7, hard-tier 75.0% against Jev 74.1%. Cost 39.5, about $0.104 estimated per 1,000. Speed uses the public-demo x2 adjustment. Qwen3.6-35B-A3B 63.8, hard 66.4%. Independent harness, author-run.
Kev family, author new-source table
79.6% vs 85.7% · JevBench 66.7
Jared Palmer. Out of domain, data Kev never trained on: Kev-8B 79.6%, Jev 85.7%. MMLU 70 versus 90, date arithmetic 60 versus 93 in the same thread. README later ships Qwen3.5 0.8B / 4B / 9B; Kev-9B new-source 0.812 / 0.837 against Jev 0.857. JevBench independent rows (Qwen3 checkpoints, commit 20fa626): kev 0.6B 66.7, kev 4B 62.2, kev 8B 58.3. Author suite is not JevBench. September 26 paired run, pinned Kev-4B against fresh jev-1.13.0: full JSON 1,216/1,253 versus 1,235/1,253. Prose parameters 1,059/1,253 versus 1,236/1,253. Loopback versus hosted HTTPS.
AstroHan, 30-task context filter
25/30 vs 22/30
Same FrontierHarness 30, same agent, deepseek-flash, one run per arm. pass@1 25 with a Jev keep-or-drop on ~2k-character tool chunks (p > 0.5), 22 without. Cost per solved task $0.113 versus $0.127. meriyah: unfiltered 1.46 million characters, 0/49 tests; filtered 49/49. Input tokens 126 million versus 123 million. Two of the 30 the filter lost. Thread only, no repo.
JevBench v1.2.8
75.4 · 534 decisions
Florian S / Benchmark Heaven. Geometric mean of Intelligence, Calibration, Speed, Cost at 25% each. Jev 1.13.0 75.4; SemIf (Qwen3.5-4B) 74.7; djev 74.3; openJev Verdict 1.4 72.5; SimpleJev Qwen3.8-27B 67.3. GPT-5.6 Luna 66.2 at rank 19 with Intelligence 96.8 and hard-tier 94.5% versus Jev 74.1%. classifier.dev fast tier 84.8 is Jev behind that host's API, listed not ranked. Self-hosted latency is adjusted ×2 plus 0.15 s. Hard items written by Opus 5 and GPT-5.6 Sol, then frozen. Independent harness, author-run.
LangChain, Jev as a judge
500/500 · $0.34 vs $28.17
Daniel Shea and Seán Roche: five frozen weather-agent traces, 100 repeats. does_pass versus one human oracle: Jev 100%, Terra 99.8%, Luna 96.4%, Claude Sonnet 4.6 80.0%. Quality-score variance: Jev 0.0000149; Luna 433×, Terra 913×, Claude 92×. Per-call cost Jev $0.00035 (0.44 s) versus Luna $0.00039 (2.50 s) and Claude $0.02811. Small corpus. Hosted Jev version not in the metadata.
Lemkin, SaaStr Connect
32% false admissions
Jason Lemkin: 600 judgments, $0.064 versus about $5 on Sonnet, about 90× cheaper. Agreement with Sonnet 59.5% on 200 borderline-weighted pairs, 67% on 200 random. Blind third-model referee on the random set: Jev 70.5%, Sonnet 77.5%. Jev 47 false admissions of 148 negatives (32%) and 2 false rejections of 52 positives. Thread only, no repo.
Aman Kumar, public sets
16k calls · filter
300 items each from Enron, SST-2, AG News, Banking77 against gpt-5.4-mini and gpt-5.6-luna. Jev leads the three short-text sets and loses Banking77 (76.0% versus 78.7% / 81.7%). Production page gates on 1,005 pages: 96 to 98% agreement, lossless reject filter, 50 to 67% fewer routing calls on replay. A 135-row extraction gate at 68.6%. Inbox 800 mails at 87.4%, no lossless threshold. Public cases in jev-eval. Pipeline rows are not.
Charly Poly, Banking77 encoders
F1 0.782 · ECE 0.105
77-way intent, human labels, three seeds. Macro-F1: Jev 0.782, ModernBERT-large-zeroshot 0.712, bart-large-mnli 0.453, gliclass-modern-base 0.381. ECE favors ModernBERT (0.081 versus 0.105). Jev assigned probability 0 to the correct label on 6.6% of decisions; the open models did that zero times in 1,155 each. At a 90% precision gate Jev auto-handled 67.5% of traffic versus 35.7%. Hosted BART p50 5,131 ms versus Jev 299 ms. Thread only, no repo.
Mathore, 200-piece Tetris
9,200 · ~300 ms · 0 illegal
Same 200-piece sequence, legal landings only. Jev 9,200 points, 75 lines, about 300 ms per move. Claude Haiku 4.5: 8,900 points, 72 lines, 1.52 s, $0.48. Gemini 3.5 Flash-Lite: 9,000 points, 75 lines, 1.13 s, $0.02. One run. Steve's clip is a separate ~400 ms / one cent per 100 pieces note.
TypeSafe workflow table
193.6x / 444.6x
Company evals on four internal workflows, the high end of the launch 20-200x / 40-400x range. Built by TypeSafe's model capabilities team. Vendor-reported. On September 30, 2026 this desk scored 32 tickets on jev-1.13.0 and gpt-6-luna: 32 of 32 each, p50 913.5 ms against 2,698.5 ms. That set did not reproduce 193.6x. The launch story has the sample and the missing dollar price.
Every, 777 judgments
0.7s · 6 of 7 defects
Mike Taylor ran 37 documents times 21 writing questions (777 answers) in under 0.7 seconds, about a quarter of a cent. Dan Shipper's later 12-passage defect check against Claude Fable 5.1: median 0.35s versus 8.83s, about 25 times faster and about 580 times cheaper, six of seven planted problems. Independent, small sample.
Bryo, 1,565 emails
96.4% vs 97.5% / 98.5%
Nikhil Mudholkar routed 1,565 German and English supplier emails into 10 categories. Jev 96.4%, Gemini 3.5 Flash-Lite 97.5%, Gemini 3.8 Flash 98.5%. Per 1,000 emails: $0.08 / $0.80 / $1.79. 737 Jev answers at 99% confidence all matched; the most confident 85.5% had zero errors on this set. 364 mails were synthetic. No attachments.
Vercel fx auto mode
5-18x vs Luna
Pranit Sharma: a command-safety classifier previously on gpt-5.6-luna was about 5 to 18 times faster and more accurate after swapping Jev. No published accuracy percentages or sample size.
News Desk Dealer, 384 headlines
24.9s · $0.19
Elvis scored a morning wire for 15 brand desks. Jev finished 384 headlines in 24.9 seconds at $0.19. Opus 5 got through four and cost $0.77 before the run was stopped. Throughput and spend, not agreement with an editor.
SREGym-Lite
24/50 vs 20/50
Jackson Clark: Codex plus gpt-5.6-luna on 10 SREGym-Lite problems, five attempts each. 20 passes without Jev, 24 with jev_plan and jev_submit at a 0.70 threshold. Two problems got worse. Authors did not measure time to diagnosis.
Laya, author table
70.1 JevBench · Banking77 0.425
Convai Innovations. JevBench independent row 70.1 versus Jev 75.3. Author T4: 32.8 ms multilingual, 39.5 ms English, one question. Author quality versus published Jev figures the README says were never measured in that repo: typed-decisions 0.766 vs 0.727 (fine-tuned), Banking77 0.425 vs 0.870 (77 vs 72 labels). Base checkpoints 0.362 / 0.342 against a 0.461 majority baseline.
Bespoke Nimble
74.8% macro vs 76.0%
324 synthetic holdout examples: Bespoke-Nimble-9B matched 292 labels, Jev 1.13.0 matched 302, untuned Qwen3.5-9B matched 215. On 13 public subsets with human labels the macro average is 74.8% against Jev at 76.0%, across 3,880 records. The September 24 checkpoint prints no new row. Labels on the 324 are unreviewed by people. Hemant's Verdict 151M encoder, on a different 337-case TypeSafe public set, scored 48.1% where Jev scored 90.8%.
Postgres JOB overlay
12% geomean
Michael Malis: Jev picking join order was about 2x slower. Estimating filters helped some IMDB queries and blew one up. Postgres plans first, Jev overrides when confident: 12% geometric-mean improvement, no dramatic regressions, planner itself slower by hundreds of milliseconds per call. Thread only, no repo.
Vogel, 1,500 emails
n = 1,500 · no accuracy
Ryan Vogel classified 1,500 personal emails on a livestream. Sample size is in the post. Precision, recall, and a comparator are not.
jeff, 1,600 public items
75.5% vs 90.5% AG News
Logan Markewich: 200 items each from eight public datasets. Sequential p50 from a laptop 151 ms on an L4 versus Jev 129 ms. About $2.6 versus about $15.6 per million single-question requests. AG News 75.5% versus 90.5%. Averages: choice accuracy 0.61 versus 0.69, noul AUROC 0.84 versus 0.975. Author-run, public labels.
SemIf, 21-question 3090 note
1.023 s vs 5.332 s
Theodore Lee: frozen Qwen3.5-4B, 21 binary criteria, median of 3. Direct logit readout 1.023 s and 0 output tokens; compact JSON array 5.332 s and 111 tokens. Argmax agreed on 18/21. Authored balanced accuracy 0.813 on 144 rows. TypeSafe subset agreement 0.845 on 102 aligned rows against a published Jev 0.883; authors did not call live Jev. JevBench is the independent row.
GitHub Next, LocalJev bake-off
76.7% short-input · 1,200 requests
Five 4-bit checkpoints on oMLX, 120 gold labels (AG News, BoolQ, SST-5), M5 Max. Short-input macro: Qwen3.6-35B-A3B 76.7%, Gemma 4 26B-A4B 75.0%, DiffusionGemma 26B-A4B 74.2%. The probabilities are prompted JSON. The bake-off does not include hosted Jev, and the authors say it is not a logit read. Long input drops every model's score.
Jev macOS Loop
6/6 · 7.39 s Finder
Josh C. Simmons: six native GUI tasks passed on Vercel and on OpenRouter. Finder batch of nine files in 7.39 seconds through Vercel, 22.69 seconds one file at a time. Median Jev round-trip about 300 ms. Small functional sample. An earlier Calculator run failed and is in the notes.
term-llm, 30 prompts
n = 30 · anecdotal
Sam Saffron's PR: 30 real session-store prompts on jev-latest, no session context, all of which should reach the model. Zero false switch_session or new_session above threshold. Navigation and status examples at or above 0.88 with untuned criteria. The author calls it anecdotal. No comparator model.