Updated
Desk
Jev vs Laya, Luna and the rest: published results side by side
Similar tests already in the stories, placed next to each other. The sets, labels, and comparators differ, so the rows are not a ranking. We did not rerun any of them. One-row write-ups live on evals. Failure modes live on limits. Decision model hallucination keeps the RAGTruth rows on their own tests. The Jev benchmark page separates JevBench from TypeSafe's own workflow test. Jev vs Laya is one of those pairs. Stories grouped by tag are on topics.
OR-decision-4B launch chart against Jev
Choice, Score, Yes/No, and Sarcasm are cells on alphaXiv's October 30 launch chart. 83.55% is the Hugging Face card's JevBench primary window, 193 of 231. Mammoth-1 Choice 85.6 on that chart is not Decision Index Full 71.24. Do not merge 83.55 with 79.7.
| Row | Choice | Score | Yes/No | Sarcasm |
|---|---|---|---|---|
| OR-decision-4B | 79.7 | 67.0 | 88.3 | 53.6 |
| Jev | 79.6 | 65.4 | 87.4 | 70.7 |
| Mammoth-1 | 85.6 | 69.1 | 91.0 | 62.1 |
| Elephant-1 | 74.7 | 62.3 | 84.7 | 60.0 |
| Joey-1 | 71.0 | 61.8 | 81.1 | 57.1 |
MiniCPM5-2B-Jev JevBench against hosted Jev
78.79% is the project's JevBench overall on 231 items. The same README table prints hosted Jev 1.13.0 at 86.6%. 86.41% and ECE 0.0307 are decision-v7, a separate author suite. Do not merge 78.79 with 86.41.
| Score | Suite | |
|---|---|---|
| MiniCPM5-2B-Jev | 78.79% | JevBench overall |
| hosted Jev 1.13.0 | 86.6% | same README JevBench table |
| MiniCPM5-2B-Jev | 86.41% | decision-v7 accuracy |
| MiniCPM5-2B-Jev ECE | 0.0307 | decision-v7 |
| TypeSafe Hosted Jev | 84.50% | decision-v7 author table |
BranchOne Colab median against nearby clocks
0.305 s is Necati Demir's A100 Colab median on ten short text questions with three choices each. 67 ms is Celeris-1 Decision's printed one-question median. 20 ms and 100 ms are hunch's published CPU clocks. Do not merge those three.
| Clock | Host | |
|---|---|---|
| BranchOne | 0.305 s | A100 Colab, ten text questions |
| Celeris-1 Decision | 67 ms | us-east-1 one question |
| hunch text | 20 ms | CPU, author claim |
| hunch images | 100 ms | CPU, author claim |
hunch clocks against nearby System One latency prints
20 ms and 100 ms are Mastracci's published hunch clocks on CPU. 38.8 ms and 209.3 ms are Cloudflare's October 1 Clef-flash and Clef medians. 35 to 60 ms is djev-run samples=1 on an RTX PRO 6000. 67 ms is Celeris-1 Decision's printed one-question median. Do not merge those four harnesses.
| Clock | Host | |
|---|---|---|
| hunch text | 20 ms | CPU, author claim |
| hunch images | 100 ms | CPU, author claim |
| Clef-flash median | 38.8 ms | October 1 blog |
| Clef median | 209.3 ms | October 1 blog |
| djev-run samples=1 | 35-60 ms | RTX PRO 6000 |
| Celeris-1 Decision | 67 ms | us-east-1 one question |
d3-ultra Full score against SR-Text and vision
70.18 is Decision Index 0.4.1 Full score, second of 229. 87.2 is the card's private SR-Text mean. 70.59 is vision full on edition 0.4. Do not merge those three.
| Score | Board | |
|---|---|---|
| Full score | 70.18 | 0.4.1 text |
| SR-Text mean | 87.2 | card private |
| Vision full | 70.59 | 0.4 vision |
d3 family on Decision Index 0.4.1
Same live space file, opened October 26. Full score is balanced_skill. Do not merge with the October 16 d3 card cell of 66.3 on edition 0.3.2.
| Checkpoint | Full | Public skill |
|---|---|---|
| d3-ultra | 70.18 | 71.57 |
| d3-turbo v1.1 | 67.31 | 71.62 |
| d3 | 67.15 | 65.45 |
| Jev 1.14 reference | 60.24 | 60.7 |
GLiDE scores, 0.2.1 self-run against 0.4.1 Full and JevBench
64.81 is Fastino's October 1 self-run of Decision Index 0.2.1. 71.5 is Full score on edition 0.4.1, opened October 26, rank 1 of 229. 76.4 is JevBench Fastino GLiDE (hosted API, 22 Oct 2026), rank 3 of 48. 30.6 is the older hosted GLiDE row. Do not merge those four.
| Score | Board | |
|---|---|---|
| 0.2.1 self-run | 64.81 | Fastino, October 1 |
| 0.4.1 Full | 71.5 | live space, rank 1 of 229 |
| JevBench hosted 22 Oct | 76.4 | rank 3 of 48 |
| JevBench older hosted | 30.6 | rank 34, $0.137 |
Jeva Table 1 against Jev-1.13.0
91.62 is BANKING77 at K=16. 78.30 is Typed Test. 0 flips is 600 Choice items. Jev-1.13.0 on those rows is 82.50, 74.10, and 17.00%. Do not merge 91.62 with 78.30.
| Slice | Jeva-4B | Jev-1.13.0 |
|---|---|---|
| BANKING77 K=16 | 91.62 | 82.50 |
| Typed Test | 78.30 | 74.10 |
| Choice flips, n=600 | 0 | 17.00% |
| Null AUROC, BANKING77 K=16 | 98.66% |
Abe 1,774 posts, Jev Noul against AI_FILTER
$0.055 against $1.87 is Abe's 1,774-row total, about 1/34. 0.9685 is AUC against AI_FILTER, not accuracy. 54 of 70 is his hand read of one split. 0.04 is this desk's jev-1.14.0 noul on the Lightdash example. 0.03 is Abe's jev-1.13.0 on that sentence.
| Jev | AI_FILTER | |
|---|---|---|
| Cost, 1,774 posts | $0.055 | $1.87 |
| Wall clock | 23.0 s | 7.1 s |
| AUC vs AI_FILTER | 0.9685 | reference |
| Lightdash example noul | 0.03 then desk 0.04 | TRUE |
mammoth-1 Full score, card 0.4 against live 0.4.1
71.24 is the Hugging Face card's Decision Index 0.4 Full score, first of 208. 68.4 is this desk's opening of edition 0.4.1 on October 23, rank 3 of 229. 62.65 is elephant-1 on the card. 60.3 is elephant-1 Full on the live chart. Do not merge those pairs.
| Score | Edition | Rank | |
|---|---|---|---|
| mammoth-1 card | 71.24 | 0.4 | 1 of 208 |
| mammoth-1 live space | 68.4 | 0.4.1 | 3 of 229 |
| elephant-1 card | 62.65 | 0.4 | tied 8 |
| elephant-1 live space | 60.3 | 0.4.1 | Full chart |
| GLiDE 1.2 live space | 71.5 | 0.4.1 | 1 of 229 |
| Jeffan-31b-1710 live space | 69.8 | 0.4.1 | 2 of 229 |
Artificial Analysis Decision Index 65, three models
Artificial Analysis, October 22, prints SubQ Snap 1.1, d1-ultra, and Elephant 9B at 65, Jev 1.14 at 57. Base Labs' replica prints mammoth-1 65.9 and elephant-1 65.2. 61.7 is Snap's October 12 Decision Index 0.2.1 cell. Do not merge 65 with 61.7 or with 65.9.
| System | Board | |
|---|---|---|
| SubQ Snap 1.1 | 65 | AA Decision Index v1.0 |
| d1-ultra | 65 | AA Decision Index v1.0 |
| Elephant 9B | 65 | AA Decision Index v1.0 |
| Jev 1.14 | 57 | AA Decision Index v1.0 |
| mammoth-1 replica | 65.9 | Base Labs AA replica |
| elephant-1 replica | 65.2 | Base Labs AA replica |
| SubQ Snap, October 12 | 61.7 | company DI 0.2.1 |
Decision Index 0.3.2 Full score against public bar
69.12 is StartLux-Decision v1.1 27B Instant Full score, README dated 2026-10-16. 63.8 is Sieve-31B Full score on the Hugging Face card. 67.59 is Drex Max public-only on Nace's product page. Full score weights private tests. The public bar does not. Do not merge those three.
| System | Edition | Kind | |
|---|---|---|---|
| StartLux v1.1 27B Instant | 69.12, first of 174 | 0.3.2 | Full |
| Drex Max | 67.59 | 0.3.2 | public only |
| Sieve-31B | 63.8 | 0.3.2 | Full |
| Sieve-31B, authors' public run | 62.52 | 0.3 | public suite |
| StartLux-Decision-27B, October 3 | 63.88 | 0.2.1 | self-run |
Datadog is_risky, 40-run mean against this desk
0.893 and 0.290 are Datadog's 40-run means on LangChain tool-risk gating. 0.94 and 0.13 are this desk's two POSTs on a shortened handbook, Date headers 08:50:46 GMT and 08:50:47 GMT October 20. Do not merge 0.13 with 0.290. Noul 82.90% to 3.88% is the 4,643-attack table, a different question.
| Run | is_risky | has_prompt_injection | n |
|---|---|---|---|
| Datadog, raw handbook | 0.893 | 40 | |
| Datadog, counterfeit in profile_description | 0.290 | 40 | |
| This desk, shortened handbook | 0.94 | 1 | |
| This desk, Datadog definition in profile_description | 0.13 | 1 | |
| Datadog Noul, raw attacks | recall 82.90% | 4,643 | |
| Datadog Noul, counterfeit prepended | recall 3.88% | 4,643 | |
| This desk, dump-the-prompt Noul | 0.99 | 1 | |
| This desk, counterfeit prepended | 0.92 | 1 |
Clairic cascade against GLM-5.3
Cloudflare blog table, one damaged-claim pipeline. Michelle's 3.5x and 77% cheaper is a live-demo post. 3.2 times and 49% cheaper are arithmetic on 19.3 / 61.7 and $6,032 / $11,740.
| Step | Clairic then Clef then GLM-5.3 | GLM-5.3 every step |
|---|---|---|
| End to end | 19.3 s | 61.7 s |
| Extract | 0.46 s, Clairic | 16.1 s |
| Decide | 0.22 s, Clef | 21.9 s |
| Per million claims | $6,032 | $11,740 |
JevBench, opened October 19
Opened October 19, 2026, about 07:20 UTC. JSON revision v1.6.29, 184 ranked open-weight systems and 38 ranked API offerings. 82.5 is Capability, first of 184 open systems. 71.76 is composite, open rank 3. H2O-Lightning-4B v1.1 is open composite rank 1 at 72.52. 70.2 is Jev 1.14.0 on the API board. Do not merge Capability with Composite.
| System | Capability | Composite | Board |
|---|---|---|---|
| basal-omni-1.0-ultra | 82.54, open rank 1 | 71.76, open rank 3 | open |
| Quyet-1.0-Large | 81.69, open rank 2 | 71.39, open rank 5 | open |
| H2O-Lightning-4B v1.1 | 75.00, open rank 10 | 72.52, open rank 1 | open |
| kettei-fat | 86.31 | 77.80, API rank 1 | api |
| Jev 1.13.0 | 77.13 | 71.49, API rank 7 | reference / api |
| Jev 1.14.0 | 75.01 | 70.20, API rank 9 | api |
Checkout ad, HiddenLayer against this desk
HiddenLayer's 91 / 9 is TypeSafe Jev on jev-ultrafast. This desk's two POSTs on October 19 used a checkout Choice written here. Date headers 07:24:57 GMT and 07:24:58 GMT. Do not merge 0.89 with 9%.
| Run | Place order | Confirm Price | Model |
|---|---|---|---|
| HiddenLayer, after ad | 9% | 91% | TypeSafe Jev API |
| HiddenLayer analog, before ad | 99.7% | in-house Qwen3.8-27B | |
| HiddenLayer analog, after ad | 19.3% | 79.9% | in-house Qwen3.8-27B |
| This desk, no ad | 1.0 | jev-1.13.0 | |
| This desk, with ad | 0.89 | 0.11 | jev-1.13.0 |
Native-audio MMAU-mini
Qwen-Audio-Jev 78.70 is the paper. ToddlerJev 68.70 is 1,000 items on its blog. AudioJev 70.90 is a reprint on that ToddlerJev table. The blog says those comparator values are published results. It did not rerun them.
| System | MMAU-mini | MMSU | MMAR |
|---|---|---|---|
| Qwen-Audio-Jev, paper | 78.70 | 84.06 | 69.90 |
| ToddlerJev, blog | 68.70 | 61.30 | 56.90 |
| AudioJev, reprinted | 70.90 | 61.70 | 55.33 |
JevBench API board, opened October 17
Opened October 17, 2026, about 01:48 UTC. Release v1.6.1, board v1.7.46, 39 ranked API offerings. kettei-fat is a v1.6.27 draw put on the v1.6.1 scale. Federica's #1 of 37 is her post. 77.8 is not ImageJevBench 75.4 and not the blog's 93.1% on 231 public items.
| System | Composite | Capability | Median |
|---|---|---|---|
| kettei-fat | 77.8, rank 1 | 86.3 | 0.20 s |
| d1-ultra | 76.1, rank 2 | 80.5 | 0.23 s |
| Sage 1.3.0 | 74.0, rank 3 | 78.6 | 0.15 s |
| kettei-chub | 73.5, rank 4 | 73.1 | 0.17 s |
| Jev 1.13.0 | 71.5, rank 7 | 77.1 | 0.24 s |
pplx-decider Decision Index 0.3, v1.1 then v1.2
v1.2 card, opened October 17, 2026. 62.8 is the October 8 space print for v1.1. 85.71% is the October 1 panel. Do not merge those prints.
| v1.1 | v1.2 joint | v1.2 one question | |
|---|---|---|---|
| Overall | 62.30 | 68.09 | 67.75 |
| Vision | 73.22 | 77.43 | — |
| Flash overall | — | 58.88 | 60.05 |
Vercel Gateway paid-team reach
September 18 blog against the October 2026 Production Index launch table. TechCrunch rounds 35.3% to 35%. 0.27%, 0.3%, and 0.15% are three spend prints.
| Window | Jev | Next named model |
|---|---|---|
| First 24 hours, September 18 post | nearly 13% | GPT-5.6 family, half that |
| First two weeks, October index table | 35.3% | Sonnet 5.5, 18.1% |
Celeris jev-bench, 3,210 items
Celeris blog, October 8, 2026, harness 0cfd276. Praveenrajus/jev-bench, 22 datasets. This table is not Florian's JevBench composite. 67 ms is us-east-1, one question.
| Model | Accuracy | Brier | 1-question median |
|---|---|---|---|
| celeris-1-decision | 0.800 | 0.272 | 67 ms |
| pplx-decider-v1.1-27b | 0.790 | 0.288 | 122 ms |
| Mercury Decide | 0.753 | 0.372 | 147 ms |
| Jev 1.13.0 | 0.731 | 0.380 | 135 ms |
| gpt-6-luna | 0.690 | 0.455 | 110 ms |
Artificial Analysis Decision Index, opened October 16
Opened October 16, 2026 UTC, language models left off, 11 of 13 models. This board is not Hugging Face Decision Index. Liquid's 112-evaluation average of 76.7 is a different table. Celeris 80.0% on 3,210 jev-bench items is a different test.
| Model | Index | Notes |
|---|---|---|
| d1-ultra | 65 | Liquid hosted, posted October 15 |
| Jev 1.14 | 57 | TypeSafe |
| GLiDE | 54 | Fastino |
| Microsoft-Decision-1 | 54 | tied with GLiDE on this print |
| Celeris-1 Decision | 52 | hosted, posted October 8 |
| hosted d1 | 50 | Liquid's earlier hosted call |
d3 Decision Index, 0.3.1 then 0.3.2
Same Hugging Face card, two openings. Public 65.5 is the public column on both prints. 66.3 is not 64.7.
| Opened | Edition | Index | Public |
|---|---|---|---|
| October 11 | 0.3.1 | 64.7 | 65.5 |
| October 16 | 0.3.2 | 66.3 | 65.5 |
Span-02 Decision Index 0.3 against the card
Respan blog, opened October 16, 2026 UTC, local rebuild of the public suite. Hugging Face card text index 63.62 is a separate print. Do not merge 61.2 with 63.62 or with d3's 64.7.
| Span-02 | Jev 1.13.0 | |
|---|---|---|
| Blog, thinking off | 61.2 | 58.0 |
| Blog, 1 s | 63.0 | — |
| Blog, no limit | 64.1 | — |
| Card, text index, 140,178 requests | 63.62 | — |
OpenRouter Armor, 492 tool calls
OpenRouter chart, October 15, 2026, 21:00 UTC. Synthetic set seeded from harness logs. This desk's 0.98 and 0.01 are two TypeSafe noul calls on October 16, not that set.
| Model | Agreement | Printed cost |
|---|---|---|
| Jev 1.13, Armor default | 90.7%, 446 of 492 | $0.0021 / call |
| Clef | 88.4%, 435 of 492 | $0.0065 / call |
| Sonnet 5 (Claude Code) | 88.2%, 434 of 492 | ~$0.022 / call est. |
| GPT-6 Luna decisions | 82.7%, 407 of 492 | $0.0061 / call |
Every-vehicle Box against TypeLLM's six-box cookbook
TypeLLM cookbook, opened October 16, 2026, frame at 14.00 s. This desk, Date headers 08:19:44 GMT to 08:19:46 GMT, jev-1.14.0. Do not merge TypeLLM's six boxes with this desk's one rectangle.
| Host | Return | Notes |
|---|---|---|
| TypeLLM cookbook, 14.00 s | six boxes, array of boxes | flying Roadster last in the 13.75 s list |
| This desk, every vehicle, n = 3 | one rectangle each, conf 0.19 / 0.17 / 0.13 | type array HTTP 400 |
| This desk, Elon frame, n = 1 | Choice six at 0.59; flying box conf 0.69 | in-air noul 0.94 |
Aviz 180-scene Box against this desk Continue n = 3
Aviz Maeir, October 15, 2026, 180 scenes, OpenAI Responses API plus strict JSON. This desk, Date headers 00:28:47 GMT to 00:28:48 GMT October 16, one Checkout PNG, n = 3. Do not merge 82% with IoU 0.93.
| Host | Jev 1.14 | Other |
|---|---|---|
| Aviz, 180 scenes | 82%, 0.21 s, $0.15 per 1k, ≥0.7 right 107 of 108 | luna 84% 1.5 s; sol 95% 3.4 s |
| This desk, Continue button, n = 3 | mean IoU 0.93, conf 0.91, noul 0.99 | jev-1.13.0 Box HTTP 400, noul 1.0 |
Four Box questions, one request against four POSTs
This desk, October 16, 2026, Date header 00:28:50 GMT. One 400 by 400 four-color PNG, image in state. Aviz billed 2,266 against 8,635 on a different file. Do not merge 4,458 with 2,266.
| Layout | Input tokens | Notes |
|---|---|---|
| Four Nouls and four Boxes, one POST | 4,458 | All Nouls 1.0, IoU 0.76 to 0.83 |
| Four separate POSTs, one color each | 6,612 | 1,653 each |
| Aviz, four targets, his file | 2,266 vs 8,635 | About 3.8 times |
Kyoto-ben 60 against this desk n = 3
Yamashita, October 15, 2026, 60 frozen phrases, unpublished criteria, three runs. This desk, Date headers 14:57:06 GMT to 14:57:09 GMT, one published sentence, a Choice written here, TypeSafe API. n = 3. Do not merge 0.833 with 29 of 30.
| Host | 1.14 | 1.13 |
|---|---|---|
| Yamashita, 60 tickets, three-run mean | 0.833, majority same on 60 | 0.817, majority same on 60 |
| This desk, うちの子ぉも, n = 3 | social_nicety, 0.69 / 0.66 / 0.64 | social_nicety, 0.33 / 0.36 / 0.34 |
Two-square PNG, blue beside red, n = 3
This desk, October 15, 2026, Date headers 14:57:24 GMT to 14:57:27 GMT. 400 by 400 PNG, blue 0.08 to 0.28, red 0.40 to 0.80 and 0.20 to 0.70. n = 3. The 06:16 red-only 0.46 mean stays a separate sample.
| Question | 1.14 | 1.13 |
|---|---|---|
| Blue Noul | 0.98, 0.98, 0.98 | 1.0, 1.0, 1.0 |
| Red Noul | 0.55, 0.55, 0.57, mean 0.56 | 0.88, 0.88, 0.84, mean 0.87 |
| Blue Box mean | 0.07 / 0.30 / 0.14 / 0.29 at 0.94 | HTTP 400 |
Turkish buyer ticket, 29 of 30 against n = 3
SosyalDijital, October 15, 2026, 30 synthetic Turkish messages through OpenRouter, one pass, unpublished prompt. This desk, Date headers 11:04:57 GMT to 11:04:59 GMT, one published sentence, a Choice written here, TypeSafe API. n = 3. Do not merge 29 of 30 with Chinese 1.30.
| Host | 1.14 | 1.13 |
|---|---|---|
| SosyalDijital, 30 tickets | 29 of 30, missed buyer as seller | 29 of 30, same miss |
| This desk, n = 3 | seller, seller, seller | seller, buyer at 0.28, seller |
Image input tokens, 850 PNG
This desk, October 15, 2026, Date headers 11:04:54 GMT to 11:04:57 GMT. Synthetic PNGs, one Noul unless noted, jev-1.14.0. Wes Bos billed $0.000096936 on a different 850 file. n = 1 per row.
| Layout | Input tokens | List price |
|---|---|---|
| 784, image in state | 1,884 | $0.000079 |
| 850, image in state | 2,238 | $0.000094 |
| 1568, image in state | 6,589 | $0.000277 |
| 850, two questions, image in state | 2,259 | $0.000095 |
| 850, two questions, reference in instructions | 4,209 | $0.000177 |
| Wes Bos, 850 photo, billed | $0.000096936 |
jev-1.14.0, Chinese payouts n = 3
This desk, October 15, 2026, Date headers 06:16:21 GMT to 06:16:25 GMT. Same Chinese payouts sentence, English questions. n = 3. The 03:03 UTC singles stay a separate sample.
| Model | Run 1 | Run 2 | Run 3 | Mean score | Mean confidence |
|---|---|---|---|---|---|
| jev-1.14.0 | 1.27 at 0.59 | 1.32 at 0.52 | 1.32 at 0.52 | 1.30 | 0.54 |
| jev-1.13.0 | 1.55 at 0.32 | 1.52 at 0.28 | 1.55 at 0.32 | 1.54 | 0.31 |
Danger-zone image, red square n = 3
This desk, October 15, 2026, Date headers 06:16:36 GMT to 06:16:40 GMT. Same 400 by 400 PNG. n = 3. The 01:22 UTC singles 0.39 and 0.90 stay a separate sample.
| Model | Run 1 | Run 2 | Run 3 | Mean |
|---|---|---|---|---|
| jev-1.14.0 Noul | 0.49 | 0.47 | 0.42 | 0.46 |
| jev-1.13.0 Noul | 0.94 | 0.94 | 0.93 | 0.94 |
| jev-1.14.0 Box conf | 0.90 | 0.92 | 0.92 | 0.91 |
jev-1.14.0, four-language payouts tickets
This desk, October 15, 2026, Date headers 03:03:53 GMT to 03:04:52 GMT. Same English questions on four translations of the docs' payouts sentence. n = 1 each. Language 1.0 on jev-1.14.0 is also 1.0 on jev-1.13.0 for these four sentences. TypeSafe's internal 60.0 chart is a different test.
| Language | 1.14.0 noul | 1.14.0 billing | 1.14.0 language | 1.14.0 frustration | 1.13.0 frustration |
|---|---|---|---|---|---|
| English | 0.95 | 0.86 at 0.79 | 1.0 | 1.02 at 0.97 | 1.03 at 0.94 |
| Chinese | 0.96 | 0.99 at 0.98 | 1.0 | 1.37 at 0.44 | 1.54 at 0.31 |
| Spanish | 0.93 | 0.96 at 0.94 | 1.0 | 1.02 at 0.97 | 1.04 at 0.94 |
| Japanese | 0.94 | 0.99 at 0.98 | 1.0 | 1.10 at 0.85 | 1.25 at 0.62 |
Jev Day aliases, payouts sentence
This desk, October 15, 2026, Date header 01:22:13 GMT. Same state, is_urgent and department. n = 1 each. TypeSafe's internal intelligence chart is a different test. 60.0 is not JevBench 71.5.
| Request | Response model | Noul | Billing | Confidence | Desk clock |
|---|---|---|---|---|---|
| jev-latest | jev-1.13.0 | 0.95 | 0.88 | 0.82 | 427 ms |
| jev-preview | jev-1.14.0 | 0.95 | 0.86 | 0.79 | 433 ms |
Danger-zone image, red square
This desk, October 15, 2026. One 400 by 400 PNG drawn here. Ground-truth square 0.40 to 0.80 width, 0.20 to 0.70 height. n = 1. imajev-4b Image JevBench is a different board.
| Call | HTTP | Noul | Box | Clock |
|---|---|---|---|---|
| jev-preview, no header | 400 | no answer | no answer | 410 ms |
| jev-1.14.0, header on | 200 | 0.39 | 0.4 / 0.8 / 0.16 / 0.7, conf 0.93 | 457 ms |
| jev-1.13.0, Noul only | 200 | 0.90 | HTTP 400 if Box | 489 ms |
TrueForge weekly client follow-up, 40 requests
Akshay Pachaar, October 12 and 13, 2026. Both arms on Claude Opus 5.5, shortlist of 10 issues, accept at 80% or 90% if closed. The 150 times line is the article's Opus rate-card estimate. 150 times $0.0015 is $0.225. That figure is a different cell from $0.005 to $0.015. On October 13, at 09:21 UTC, he wrote that 40 is too small to establish how reliably confidence routes errors to review. This desk's two calls are a request written here, n = 1 each.
| Arm | Correct | Second look | Wrong caught | Decision cost |
|---|---|---|---|---|
| weekly-client-follow-up | 37 of 40 | 16 | 5 | $0.0015 |
| weekly-client-follow-up-no-jev | 37 of 40 | 5 | 0 | $0.005 to $0.015 |
| This desk, intended shortlist | ISSUE-1406 at 1.0 | would accept at 80% | n = 1 | $0.000035 list |
| This desk, trap shortlist | ISSUE-844 at 0.55 | confidence 0.51 | n = 1 | $0.000032 list |
JevEval, Paris weather Tool Faithfulness
This desk, October 14, 2026, Date header 01:58:11 GMT. Two POSTs, jev-latest, response jev-1.13.0. Questions and state follow DeepEval's published Tool Faithfulness example. Credits applied here after the response. The 0.461 column is the docs walkthrough, which calls its probabilities illustrative. This desk did not install deepeval. n = 1 per row.
| Answer | Weighted mean | strict_mode | Min confidence | Input tokens | Desk clock |
|---|---|---|---|---|---|
| Invented breeze and 40% humidity | 0.254 | 0 | 0.26 | 545 | 478 ms |
| 18°C and sunny only | 0.901 | 1 | 0.64 | 533 | 478 ms |
| Docs walkthrough (illustrative) | 0.461 | not printed | 0.40 | not a live call | not a live call |
Pairwise Choice, two lists of eight
This desk, October 12, 2026. Eight passages per list, the same query, jev-1.13.0. Spearman is against this desk's intended order H, G, F, E, D, C, B, A. The Stripe shortlist is all Stripe keys or webhooks. The mixed list adds Lisbon weather and a PostgreSQL password reset. n = 8. These texts are not a retrieval corpus.
| List | Numeric 0-9 | Four-level Score | Noul | Pairwise BT |
|---|---|---|---|---|
| Stripe shortlist | 0.690 | 0.738 | 0.357 | 0.548 |
| Mixed list | 0.929 | 0.929 | 0.738 | 0.857 |
Quyet-1.0-Large, frozen JevBench v1.6.0
quyet.ai, opened October 12, 2026. One frozen release, v1.6.0, 92 systems. The October 8 rank line says official composite 3 at the same Capability 81.7. H2O's 71.4 is revision v1.7.17. Jev's later capability of 77.1 is v1.6.1. We did not reopen the frozen file.
| System | Official score | Capability | Intelligence |
|---|---|---|---|
| Quyet-1.0-Large | 71.39 | 81.7 | 73.4 |
| Jev 1.13 | 71.09 | 76.5 | 62.4 |
| Clef | 16.75 | 75.3 | 59.1 |
Stripe ticket, one call each
This desk. Snap is October 13, 2026, about 03:00 UTC, POST https://api.subq.ai/v1/systemone, model subq-snap-preview. Jev and Decisions are the October 6 single calls on the OpenAI Jev page. The Snap noul questions included true and false criteria the other two hosts did not. One call is not a ranking. The October 7 and October 8 ten-call medians stay on that page.
| Host | Urgency noul | Choice | Score | Input tokens | Desk clock |
|---|---|---|---|---|---|
| SubQ Snap, October 13 UTC | 0.98 | technical 0.90 | 0.98 | 119 then 235 | 1,195 ms then 2,291 ms |
| Jev, October 6 | 0.98 | technical 0.74 | 1.0 | 302 then 425 | 679 ms then 649 ms |
| OpenAI Decisions, October 6 | 0.99 | technical 0.99 | 1.0 | 182 then 444 | 1,596 ms then 476 ms |
SubQ Snap, company Decision Index 0.2.1
Subquadratic blog, October 12, 2026. Company tables. The Jev cell on those tables is 57.9. The September 28 board file still lists Jev at 57.91. Opened October 13, 2026 UTC, the live Decision Index space headers v0.3.2, 149 models, Jev 60.1, updated 2026-10-12. A search of that page for Snap returned no match. We did not rerun the suites.
| Check | Snap | Jev cell | Caveat |
|---|---|---|---|
| Decision Index 0.2.1 public suite | 61.7 | 57.9 | All requests under about 34K tokens. The paragraph also prints 61.70 and 57.90. Blink's self-run of this edition is 57.48 against 57.91. |
| 16 enterprise Decision Index tasks | 67.54 | 61.6 | Table average, lead on 13 of 16. The TL;DR prints 67.6. The homepage prints 67.5. |
| Nimble Suite 13 | 82.9% | 76.0% | 3,880 examples. Ollama later left Jev at 76.0% on a table where Nimble is 75.7%. The 324-row holdout stays 90.12% against 93.21%. |
SubQ Snap, company long-context tables
Same October 12 blog. Decision RULER is a four-option rewrite of 11 RULER tasks. Decision LCR is a rewrite of Artificial Analysis AA-LCR. The blog says LCR scores cannot be compared with published AA-LCR results. We did not rerun them.
| Check | Snap | Jev cell | Caveat |
|---|---|---|---|
| Decision RULER, 32K | 99.8 | 94.1 | Jev has no printed cell past 32K. Snap's later cells are 64K 99.8, 128K 99.5, 256K 99.8. |
| Decision RULER, 1M tokens | 96.3 | unprinted | TL;DR cell. The teaser alt lists 99.3 and 96.3 after the 256K row. |
| Decision LCR, 72K-115K | 74.0 one-shot | 48.0 chunked | 100 questions became 118 decisions, three option orders, credit only if every part is correct. |
Blink v0.3, Decision Index 0.2.1
Pixilab card, opened October 12, 2026. One self-run of edition 0.2.1. The card says Blink's row is not a board submission. The gap from 57.48 to 57.91 is 0.43, outside the kit's 0.25 tie band. Edition 0.3's 60.1 is a different board. We did not run the weights.
| Model | Index | Language | Tools |
|---|---|---|---|
| Jev, board cell | 57.91 | 62.0 | 75.1 |
| Blink v0.3 | 57.48 | 64.3 | 70.0 |
| Surogate Rune v3 | 57.44 | 63.1 | 71.2 |
| Blink v0.2 | 55.97 | 60.4 | 69.3 |
TypeLLM, one public JevBench subset
TypeLLM summary files, September 23, 2026. One checkpoint, RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead. Thinking had no token budget. The homepage's 86.58% for Jev 1.13.0 is an external row the eval README attributes to Open-Jev. This desk's JevBench composite and Drex's 199 of 231 are other tables. We did not rerun the files.
| Tier | No thinking | Thinking |
|---|---|---|
| All 231 | 195 | 228 |
| Hard, of 111 | 76 | 109 |
| Easy, of 48 | 48 | 47 |
| Original, of 72 | 71 | 72 |
Microsoft-Decision-1, October 9 speed lines
Achint Srivastava, Command Line, October 9, 2026. One comparison run by Microsoft. The post says Microsoft-Decision-1 was the fastest model measured. It does not print milliseconds, and it does not print the accuracy it calls the highest. The appendix names Quyet-1.0-Large, Surogate Rune 26B-A4B, GPT-6 Luna Decisions, deck-31B, H2O-Lightning-4B, and Strands-Decider 2B. It does not name Jev. OpenRouter's Azure P50 of 0.18 seconds, read October 10, is a different clock. JevBench's API page that day had no Microsoft-Decision-1 string. We did not rerun either measurement.
| Comparison | What the post prints |
|---|---|
| Versus Quyet-1.0-Large | 4.5 times quicker |
| Versus GPT-6 Sol | 35 times quicker |
| P50 versus GPT-6 Sol | about 35 times faster |
| Eight perturbations | decision flips on 1.3% |
| Paraphrase, reverse, or shuffle | zero flips |
Step-Jev, one BrowseComp-Plus seed
bandr-ai/bandits, step-rl-results.md, status 2026-10-07. Qwen3-4B-Instruct-2507, 60 held-out questions, eight rollouts each. Accuracy is a strict string match. Jev final ran 8 of 10 updates. The other arms ran 10. On October 9, 2026 this desk scored the repository's test states with jev-1.13.0. The no-step state was incorrect at 1.0. The five-step fixture was correct at 0.05. A Moon-landing question written here scored 1965 with no search incorrect at 1.0. Those states are not this table. real_jev_gate.py's rollout files are not in the repository. We did not rerun the table.
| Arm | Accuracy | Tool calls | Repeats |
|---|---|---|---|
| No RL | 15.6% | 3.6 | 4% |
| String-match answer key | 22.9% | 4.1 | 8% |
| Outcome Jev only | 5.4% | 10.5 | 15% |
| Outcome Jev plus step Jev | 19.6% | 6.5 | 11% |
| Hosted Jev, final only | 0% | 0.0 | 0% |
| Hosted Jev, final and step | 0% | 0.3 | 0% |
Arena, typesafe/jev-router against DeepSeek V4.1 Flash
Arena, October 7, 2026. One Agent Arena run, more than 4,700 sessions. The DeepSeek column is the model the router picked most. On Steerability and Praise vs Complaint the thread prints the router first. On Confirmed Success, Bash Recovery, and Tool Hallucination it prints DeepSeek first and says DeepSeek is ahead. The thread does not print DeepSeek's net score or its dollar cost. Opus 5.5 (High) at +10.48 is a steerability neighbor and is not a column here. The September 25 count of 237 of 423 is a different test. We did not rerun it.
| Measure | typesafe/jev-router | DeepSeek V4.1 Flash |
|---|---|---|
| Net, if listed | +3.6% | |
| Median cost per task | $0.155 | |
| Steerability | +10.0% | -0.2% |
| Praise vs Complaint | +6.3% | +2.1% |
| Confirmed Success | +2.2% | +8.3% |
| Bash Recovery | -0.7% | +7.8% |
| Tool Hallucination | +0.2% | +0.4% |
| Median latency | 6.18 s | 3.64 s |
| P90 latency | 30.59 s | 14.26 s |
Decision Index 0.3, full score, October 7
Opened October 8, 2026, the Decision Index space still headers v0.3, 115 models, updated 2026-10-07. The head-to-head panel still lists Perplexity Decider v1.1 at 62.8 and Jev as a reference at 60.1. Hugging Face Decision Index space, opened October 7, 2026. Header: v0.3, 112 models, Jev jev-1.13.0, updated 2026-10-06. Jev is a reference on the full-score table. The v1.1 card's own table prints 61.56 against Jev at 57.9, a different reading from 62.8. We did not rerun the board.
| Model | Full score | Rank |
|---|---|---|
| Jev 1.13.0 | 60.1 | reference |
| Perplexity Decider v1.1 (27B) | 62.8 | 1 |
| Fastino GLiDE no-thinking (28B) | 60.2 | 2 |
| Torchcast Decision 27B | 59.9 | 3 |
Drex DLM on Decision Index 0.2
nace-ai/drex-dlm, opened October 7, 2026. The post calls the release Drex 1.1. The card title is Drex DLM. The section is labelled Decision Index 0.2 and says the other five rows are the top of that board. The kit's Jev cell is 51.67. The hosted Drex 1.5 row on edition 0.2.1 is 58.28. A search of the 0.3 space for Drex returned no row. We did not rerun it.
| Model | Index |
|---|---|
| Drex DLM | 52.31 |
| Decider chat Gemma-4-31B | 51.93 |
| Jev | 51.67 |
| AutoJev-27B | 50.94 |
| Jebadiah 27B | 50.32 |
| simple-jev, Qwen3.8-27B | 50.21 |
HiringCafe, Spearman, October 7
Hamed Nilforoshan, October 7, 2026. The post's summary line says the OpenAI call is twice as expensive and 5 to 10 percent worse. The printed figures are Spearman correlations. The thread does not print the pair count, a price, or a latency. We did not rerun it.
| Task | Jev | OpenAI | Gemini 3.1 FL |
|---|---|---|---|
| Query to job | 0.74 | 0.71 | 0.63 |
| Resume to job | 0.71 | 0.67 | 0.61 |
PolicyLM custom policy, 0.5 cutoff
musubilabs/policylm-1.7b, opened October 7, 2026. Accuracy at a 0.5 cutoff. The card says every set except OR-Bench was also used in development. The table has no Jev column. PolicyLM returns a score from 0 to 1 for each category, up to 16 in one pass. We did not rerun it.
| Model | Accuracy | Policy edits followed | H100 median |
|---|---|---|---|
| PolicyLM-1.7B | 0.842 | 0.528 | 22 ms |
| gpt-oss-safeguard-20B | 0.909 | 0.716 | 349 ms |
| CoPE-B-A4B | 0.829 | ||
| Granite Guardian 4.1-8B | 0.722 |
Tev1 on Ollama's 3,880 decisions
Ollama's tev1 page, opened October 7, 2026. The Jev cell is Bespoke's published API run, printed as 76.0%. Together's 880 of 1,000 development set is not this table. We did not run the models.
| Model | Accuracy |
|---|---|
| Tev1 4B | 73.3% |
| Tev1 0.8B | 63.5% |
| Nimble 9B | 75.7% |
| Jev 1.13 | 76.0% |
ForkScape's 450 labelled decisions
ForkScape, report dated October 5. Hand labels, graded by code. fsdecide is a 149M ModernBERT. jev-preview stays within about a point of jev-latest. The general file is the authors' JevBench-mini, 1,248 requests, not Florian's board. We did not rerun it.
| Suite | n | fsdecide | jev-latest |
|---|---|---|---|
| Triage | 80 | 97.5% (91.3% to 99.3%) | 92.5% (84.6% to 96.5%) |
| Review | 80 | 91.3% (83.0% to 95.7%) | 56.3% (45.3% to 66.6%) |
| Browse | 160 | 88.8% (82.9% to 92.8%) | 81.3% (74.5% to 86.5%) |
| Router | 70 | 95.7% (88.1% to 98.5%) | 100% (94.8% to 100%) |
| Command intent | 60 | 100% (94.0% to 100%) | 78.3% (66.4% to 86.9%) |
| General-set accuracy | 1,248 requests | 60.5% | 90.5% |
Decision 2.0 cards, opened October 5
vLLM Semantic Router, posted October 3. Apache-2.0. Index cells are the authors' 0.2.1 reproduction. The cards do not print a Jev row. The September 28 board's Jev cell is 57.91. Transfer is median macro-F1 over 15 tasks, times 100. The median is one question on an unnamed GPU. We did not run the weights.
| Model | Parameters | JevArena | Transfer | Index | Median |
|---|---|---|---|---|---|
| Kai | 0.60B | 48.6 | 45.9 | 16.3 | 4.9 ms |
| Eos | 0.75B | 53.9 | 50.3 | 20.1 | 6.0 ms |
| Sol | 1.88B | 52.1 | 51.3 | 29.5 | 7.2 ms |
| Nox | 4.21B | 63.6 | 52.3 | 43.8 | 12.9 ms |
| Lux | 7.94B | 68.1 | 56.2 | 46.3 | 18.4 ms |
| Vega | 29.37B | 74.0 | 58.7 | 56.5 | 71.4 ms |
Vela 2.0 family index, 9B evaluation file
KR Labs and vLLM Semantic Router, file opened October 8. Edition 0.2.1, 38 benchmarks. The Vela cells in this column use noul_calibration. The default column is the card or the blog with that option off. Lux, Nox, and Eos are the harness bases printed in the same file, not the October 5 card cells. The file does not print a Jev row. The 0.3B index cell is blank. We did not run the weights.
| Size | Calibration on | Default | Harness base |
|---|---|---|---|
| 0.8B | 16.01 | 16.01 | Eos 20.14 |
| 4B | 31.91 | 31.63 | Nox 42.55 |
| 9B | 41.63 | 41.09 | Lux 46.23 |
JEV-27B-VL, camera and browser
AutoTrust card, section dated October 3. Pick-and-place is 20 scenes. Browser tasks are 60. The six-benchmark mean is a September 27 run, and its JevBench column is not Florian's board. We did not run the model.
| Check | JEV-27B-VL | JEV-9B | GEV-26B-Decide |
|---|---|---|---|
| Pick and place | 75% | 50% | 40% |
| Browser, with element text | 95% | 95% | 95% |
| Browser, boxes only | 10% | 37% | 15% |
| Robot decision | 239 ms | 163 ms | 61 ms |
Sexual text, OpenAI moderation eval
Omar Hernandez, jevmod, remeasured September 23. Human sexual labels on 1,680 messages, mapped to the nsfw question. The question also names graphic gore. F1 at default uses each system's shipped line. jevmod's is 0.80. We did not rerun it. ZeroNSFW's image cascade is a different job and prints no labelled accuracy.
| System | AUROC | F1 at default | Best F1 |
|---|---|---|---|
| jevmod (Jev) | 0.984 | 0.866 at 0.80 | 0.877 at 0.85 |
| Llama Guard 3 8B | 0.843 | 0.780 | 0.782 |
| ShieldGemma 2B | 0.968 | 0.800 | 0.851 at 0.90 |
| toxic-bert | 0.876 | 0.549 | 0.574 |
Perplexity's 7,210-row panel
Perplexity, October 1. Fixed panel, September 2026. Columns are jev-1.13.0, Qwen3.8-27B, and pplx-decider-v1-27b. We did not rerun it.
| Benchmark | Rows | Jev | Qwen3.8-27B | pplx-decider |
|---|---|---|---|---|
| WinoGrande | 1000 | 90.70% | 73.10% | 83.30% |
| FinancialPhraseBank | 999 | 76.98% | 75.68% | 84.18% |
| RAGTruth | 1500 | 77.27% | 61.53% | 88.80% |
| JudgeBench | 350 | 78.57% | 68.86% | 78.29% |
| BBH | 750 | 94.27% | 72.80% | 82.80% |
| JevBench public hard | 101 | 73.27% | 72.28% | 70.30% |
| TabFact | 500 | 89.80% | 78.60% | 90.60% |
| Circa | 500 | 84.60% | 87.00% | 89.20% |
| Belebele | 500 | 95.00% | 93.20% | 94.00% |
| TruthfulQA binary | 500 | 92.00% | 82.80% | 85.40% |
| ContractNLI | 510 | 77.45% | 80.78% | 80.78% |
| Overall | 7210 | 84.51% | 74.76% | 85.71% |
FireRed Elite Four, one run
Aravind Srinivas, October 4. One cleared run of Pokémon FireRed's Elite Four and Champion on Perplexity's Decisions API. The post prints no Jev cell and no second model. The attachment is a video. We did not rerun it.
| Stat | Posted figure |
|---|---|
| Median API response | 592 ms |
| p95 | 987 ms |
| Responses under one second | 96.4% |
| Live calls | 137 |
| Estimated inference cost | $0.028 |
Decision Index 0.2.1, October self-runs
Each row is the author's own scoring, or a chart that says so. The September 28 board file still lists Jev at 57.91. The kit's tie band is 0.25. We did not re-download that file for these names.
| Who | Figure | Jev cell they cite | Caveat |
|---|---|---|---|
| SubQ Snap, October 12 blog | 61.7 | 57.9 | Company-reported edition 0.2.1. Live space opened October 13, 2026 UTC is v0.3.2, Jev 60.1. A search found no Snap row. |
| GLiDE, Fastino blog | 64.81, 31 of 38 | 57.91 published | Self-run. Submissions paused. Not a board row. Tools +8.4, Knowledge +11.5. |
| StartLux-Decision-27B | 63.88, 31 of 38 | 57.91 published | Self-run. Not a board row. Knowledge 44.3 against 51.4. Train splits of 14 benchmarks were in the training data. |
| GEV-26B-Decide | adaptive 62.48, raw 70.66 | 57.91 on the board | Self-run. Adaptive Knowledge median 13.4 seconds, over the 1,000 ms cap. |
| Fastino chart, October 3 | Clef 61.2, Clef-flash 57.1, pplx 56.39 | 57.91, public leaderboard | Caption says Cloudflare scores are self-reported and Fastino ran Perplexity. Axis starts at 56. |
Werdykt v1, Polish and English
Remek Kinas, export dated October 4. Hidden questions, 5,000 decisions, 10 categories in each language. Jev's row has no protocol deviation. The basal SGLang rows raise the context limit to 32,768 tokens. Open-model cost is H100 time at $3.95 an hour. We did not rerun it.
| Model | PL+EN | 95% interval | p50 | USD / 1,000 |
|---|---|---|---|---|
| Jev 1.13.0, API | 81.64% | 80.70% to 82.58% | 286.4 ms | 0.0710 |
| basal-1.5-max, 11B | 77.28% | 76.24% to 78.28% | 67.1 ms | 0.1081 |
| basal-1.5, 4.5B | 72.12% | 71.02% to 73.26% | 33.8 ms | 0.0477 |
| basal-1.0, 4.5B | 66.04% | 64.70% to 67.18% | 26.5 ms | 0.1353 |
| basal-1.5-mini, 1.5B | 60.66% | 59.40% to 61.92% | 14.2 ms | 0.0535 |
StreamDecisionBench, log-AUC
Jacob Lin, October 2. Normalized log-AUC over 0.5 to 8 seconds, read off the chart image. The live HTML did not render. The arXiv table is a different scoring.
| Model | Kind | Score | Caveat |
|---|---|---|---|
| Kev-27B | open weight | 62.15 | Thread says this is the only open-weight row near Jev. |
| Jev | API | 59.63 | Not Palmer's 79.6% against 85.7%. |
| GPT-5.6-Terra, none | API | 54.11 | Reasoning set to none. |
| GPT-5.6-Terra, low | API | 48.04 | |
| GPT-5.6-Luna, low | API | 45.26 | |
| GPT-6-Astra, low | API | 45.15 | |
| Cloudflare Clef | API | 30.96 | Not the blog median of 209.3 ms, and not 61.2. |
| GPT-5.6-Luna, none | API | 30.67 | |
| Kev-9B | open weight | 28.76 | Tested after 4B, before 27B. |
| Kev-4B | open weight | 21.22 | |
| DJev / DiffusionGemma | open weight | 20.89 | |
| Clef Flash | API | 19.69 | |
| Qwen3.5-4B, direct logits | open weight | 15.61 | |
| Bespoke Nimble-9B | open weight | 14.43 | Not 292/324 against 302/324. |
| Laya, typed | open weight | 1.46 | |
| Laya, English | open weight | 0.42 | |
| Laya, multilingual | open weight | 0.21 |
OSWorld-50, one printed callout
Pranjal Aggarwal, October 2. Only the Jev callout has printed coordinates. The annotation and the opening post use different ratios.
| Line | What it says | Caveat |
|---|---|---|
| Chart callout | 9.8% success, 125 seconds a task | 50 OSWorld tasks. |
| Chart annotation | As fast as Astra xhigh, 9x lower accuracy | Printed on the plot. |
| Opening post | 1.5x slower than Opus-5.5-xHigh, 10x lower accuracy, 5x more steps | Author's framing. Not the same sentence as the annotation. |
| Thread | Model call under 0.2 seconds, parser about 500 ms a step | Jev is text only on this run. |
Open-Jev 2B, 64-step pilot
Zefan Cai, October 2. Released 2B checkpoint, synthetic controls, one H200. The 231-task card stays 197/231. We did not rerun the pilot.
| Slice | Before | After | Caveat |
|---|---|---|---|
| Test, 128 rows | 84/128, 65.6% | 99/128, 77.3% | Argmax. Temperature does not move this count. |
| Out of distribution, 128 | 83/128, 64.8% | 95/128, 74.2% | Same records, frozen backbone. |
| Test NLL / Brier / ECE | 0.8900 / 0.4903 / 0.1353 | 0.6046 / 0.3266 / 0.1088 | Includes the new temperature. |
| Timeline, out of distribution | 58.3% | 54.2% | This family fell. Authorization rose from 75% to 100%. |
WrangleBench, matches per 200
John Crown, September 30. Two bars on one chart. The post says 30.64 percent more matches at 75 percent of the previous reranking cost, and that the full benchmark is still to come. We did not rerun the searches.
| Check | With Jev | Without Jev | Caveat |
|---|---|---|---|
| Perfect matches / 200 | 137.3 | 105.1 | Averages. Query count not printed. |
| Cost | 75 percent of the previous ranker | the previous ranker, unnamed | No dollar figure on the chart or in the post. |
MMAI Gym, five CYP assays
Mikołaj Mizera, October 2. 12,563 molecules. Gradient boosting with no calibration step, against Jev 1.13. Order is 1A2, 2C9, 2C19, 2D6, 3A4. We did not rerun the assays.
| Assay | Ensemble AUROC | Jev AUROC | Balanced accuracy |
|---|---|---|---|
| 1A2 | 0.94 | 0.76 | 0.87 against 0.65 |
| 2C9 | 0.89 | 0.73 | 0.79 against 0.55 |
| 2C19 | 0.89 | 0.60 | 0.83 against 0.49 |
| 2D6 | 0.89 | 0.78 | 0.74 against 0.67 |
| 3A4 | 0.90 | 0.67 | 0.81 against 0.57 |
You.com risk sweeps
Edward Irby, October 1. Three escalated sweeps on the invoice, then a paired threshold run and a judge swap. Jev is priced at $0.042 per million input tokens. We did not run a sweep.
| Check | Jev arm | Qwen arm | Caveat |
|---|---|---|---|
| Invoice, three sweeps | Jev $0.0025 | Qwen inference $0.0619 | You.com $0.37. Total $0.43, $0.14 a sweep. |
| Escalations | 9 of 9 | 6 of 11 attempts | Gulf triage 0.35, then 0.68, then 0.50. |
| Blind mean, 1 to 5 | 4.25 (n=9) | 4.25 (n=6) | Qwen higher on factual grounding, 4.83 against 4.44. |
| Threshold 0.7 | Gulf 0.58, stopped | 0.5 arm had marked Gulf critical | 14 searches against 36. Kept 0.5. |
Bud Decision Studio, held-out training
Bud Ecosystem, October 2. The README's NVIDIA GB10 table. Example counts are not printed. General decisions is a delta whose unit is not defined in the table. We did not install the app.
| Model | Task | Before to after | General decisions |
|---|---|---|---|
| Julia 1 | support tickets | 52% to 91% | -0.4 |
| Laya Multilingual | business workflows | 34% to 62% | +12.7 |
| CLM 8B | business workflows | 39% to 68% | +11.2 |
| Jev-Omni | business workflows | 62% to 77% | 0.0 |
| Kev 4B | policy topics | 77% to 82% | +0.2 |
DSPy chart, Jev to Luna
Slide in Nathan LeClaire's September 30 thread. 100 test pairs. The marked policy was frozen on 30 validation pairs. The Jev-only end of the curve has no numeric callout. We did not run it.
| Point | Accuracy | Cost per 1,000 | Caveat |
|---|---|---|---|
| Standalone Luna | 92.5% | $0.746 | Printed callout. |
| Live threshold about 0.78 | 88.0% | $0.583 | Printed callout. |
| Jev only | not printed on the slide | not printed on the slide | Label only. |
| Shell gate in the patterns repo | deny below 0.2, ask below 0.6 | jev-1.13.0 | Hand-set. Fitted value not printed. |
Strands Decider, public 231 on their harness
strands-labs/strands-decider, opened October 2. v19 is the reference recipe at the 3072-token window. The rank is the September 25 v1.4.2 board, 89 systems. The file does not print Jev's overall count on the 231. Open-Jev's README prints 200 of 231, and Drex's chart prints 201 of 231. Those are other published cells. We did not run the model.
| Check | v19 | Other cell in the file | Caveat |
|---|---|---|---|
| Public tasks | 167/231 · 0.723 | Jev overall not printed | Brier 0.342. ECE 0.052. 4096 tokens prints 168. |
| Tiers | 1.000 / 0.875 / 0.505 | 48 / 72 / 111 tasks | Their split. Hard is 0.505. |
| 2B class, Sep 25 board | 3rd of 33 · 50th of 89 | decision-2b 0.753 · system-one-open 0.732 | Board decider-2b is 0.710, 164/231. Their v11 is 175/231. |
| Latency | RTX 3090 median 115 ms | p95 299 ms · M3 Pro 153 ms under 300 tokens | The blog opening says tens of milliseconds. Figure 3 is v18. |
| Changed question | same answer about 94% | temporal_numeric 0.267 · file says Jev 0.27 | State and options held fixed. |
HydroJEV, simulated alarm screens
arXiv:2610.02048, submitted October 1. jev-1.13.0 on stochastic EPANET windows. R1 is a hand-written rule tree. R2 is a classifier trained on 200 labels. The gate accepts a benign Jev verdict only when R1 agrees. The windows are simulated. We did not rerun the calls.
| Check | Jev | Other | Caveat |
|---|---|---|---|
| In distribution, mean | 0.63 | R1 0.59 · R2 0.91 | Four sets: 0.620, 0.628, 0.628, 0.644. No single R1 gap was significant. |
| Leave one subtype out | +0.361 to +0.417 | R2 near chance | All four gaps survived Holm correction. |
| New event families | mean 0.51 | R1 0.50 · R2 0.69 | The paper says the two training-free methods are close. |
| Round 4 cascade | gate 69/200 and 76/200 | glm cascade 0.745 / 0.565 · glm alone 0.735 / 0.549 | E4c: 32 of 76 accepts wrong, including 21 attacks. |
| Time | median compute 1.215 s | glm one at a time 30.6 to 31.7 s | 2,722 calls before round 4. Wall median 1.463 s. |
DimABSA, SemEval valence and arousal
arXiv:2609.35293, submitted September 28. Frozen jev-1.13.0 and 488 coefficients fit on CPU. Task 1 is micro RMSE, lower better. Tasks 2 and 3 are continuous F1. We did not rerun the scorer.
| System | Task 1 RMSE | Task 2 | Task 3 | Caveat |
|---|---|---|---|---|
| This pipeline | 1.0645 | 52.09 | 44.06 | Lowest Task 1 among 14 full-coverage teams. 0.0018 under PAI, untested. |
| PAI | 1.0663 | 57.73 | not printed here | Participant aggregate reconstructed from Yu et al. 2026. |
| Llama-3.3-70B | 2.5683 | 46.40 | 38.62 | 4-bit QLoRA baseline in the same table. |
| GPT-OSS-120B | 1.2362 | 45.71 | 37.27 | 4-bit QLoRA baseline. |
| Takoyaki | not printed here | 56.20 | 48.03 | Generates text. About four points above this pipeline. |
| Raw Jev, before the fit | 2.0731 | not printed here | not printed here | Nine demonstrations. The ridge fit brings Task 1 to 1.0645. |
Two filings, tree search
VectifyAI/jev-doc-search, opened October 2. Two questions are the whole printed set. jevsearch's Hit@1 and JEV Search's nDCG are different tests. We did not run the scripts.
| Filing | Pages | Pages kept | Answer page | Caveat |
|---|---|---|---|---|
| NVIDIA fiscal 2026 10-K | 93 · about 76k tokens | p37, p51 | p51 | Marked correct. Flat Choice would pass the 32k token stop. |
| Citigroup 2025 10-K | 318 | p12, p16, p134, p135 | p134 | Marked correct. A Choice stops at 255 options. |
Clef and Clef-flash, Cloudflare's October 1 tables
Cloudflare's blog and model cards, opened October 2. The latency and workflow cells are the blog. GPQA, BBH, MMLU-Pro, and RAGTruth are the model card's one-decimal run of Decision Index 0.2.1. The Jev price is TypeSafe's models page as recorded on Access, not a cell in Cloudflare's table. We did not call either host for this group.
| Check | Clef | Clef-flash | Jev cell | Caveat |
|---|---|---|---|---|
| Median latency | 209.3 ms | 38.8 ms | 524.1 ms | Laya's median on the same row is 5.8 ms. 524.1 / 38.8 is about 13.5. |
| p95 latency | 238.6 ms | 122.4 ms | 536.0 ms | Laya's p95 on that row is 222.5 ms, above Clef-flash. |
| BANKING77 macro-F1 | 94.20 | 90.93 | 79.74 | Not Laya's 0.425 vs 0.870 table, and not Charly Poly's F1. |
| When2Call | 72.37 | 65.58 | 80.97 | Blog row. The card rounds the same run to 72.4, 65.6, 81.0. |
| GPQA Diamond | 48.0 | 51.0 | 78.3 | Model card. One of eleven rows where the card marks Jev highest. |
| BBH | 73.7 | 68.9 | 92.9 | Model card. |
| MMLU-Pro | 65.9 | 65.3 | 82.7 | Model card. |
| RAGTruth F1 | 79.4 | 35.6 | 76.5 | Model card. The small model drops here. |
| Invoice, exact actions | 64.7 | 57.1 | 61.8 | Blog workflow table. Primary action on the card is 86.2, 73.3, 83.1. |
| Agent trace | 68.5 | 69.8 | 71.6 | The workflow row Jev leads. The blog's 3 of 4 does not include this row. |
| Input price / million | $0.24 | $0.09 | $0.042 | Jev output is free on TypeSafe's page. The Clef cards print no output price. |
Clef-omni, October 9 ten-row table
Cloudflare's October 9 blog. Clef, Clef-flash, and Jev cells match the October 1 print. The October 1 price row above still lists Clef-flash at $0.09. This table's price row is the October 9 blog. The 1.5 second video line is not a cell in the ten benchmarks. We did not call the API.
| Check | Clef-omni | Clef | Clef-flash | Jev cell |
|---|---|---|---|---|
| BANKING77 macro-F1 | 94.8 | 94.20 | 90.93 | 79.74 |
| CLINC150+OOS macro-F1 | 97.7 | 97.43 | 66.77 | 89.27 |
| When2Call | 63.3 | 72.37 | 65.58 | 80.97 |
| Input price / million | $0.15 | $0.24 | $0.038 | $0.042 |
JevBench API composite, October 11
Benchmark Heaven API board, opened October 11, 2026. One board, release v1.6.1, 27 hosted systems. Capability and the $ per 1,000 column are other prints on the same rows. H2O's 72.5 is the October 7 open-weights composite, a different table. We did not rerun the suite.
| Call | Composite | Capability | $ / 1,000 |
|---|---|---|---|
| Sage 1.3.0 | 74.0 | 78.6 | est. $0.025 |
| d1 | 73.0 | 74.4 | $0.017 |
| Mercury Decide | 72.4 | 73.6 | est. $0.018 |
| Jev 1.13.0 | 71.5 | 77.1 | $0.032 |
| Microsoft-Decision-1 | 69.1 | 70.8 | $0.019 |
H2O-Lightning-4B, October 7 composite
H2O blog, October 7, 2026. Open-weights composite, revision v1.7.17. Jev is the unranked reference on that table. The API board's 74.0 for Sage is a different ranking. We did not run the weights.
| System | Composite | Intelligence | Calibration |
|---|---|---|---|
| H2O-Lightning-4B v1.1 | 72.5 | 60.0 | 90.0 |
| Jev 1.13.0, reference | 71.5 | 63.6 | 90.6 |
| Quyet-1.0-Large | 71.4 | 73.4 | 90.0 |
OpenBles Firewall 1, two held-out splits
README, opened October 11, 2026. The agent split and the classic split are different files. Jev's rows were run October 2 as jev-1.13.0. We did not run the weights.
| Split | OpenBles Firewall 1 | Jev 1.13.0 |
|---|---|---|
| Agent attacks, of 120 | 119 caught, 0 look-alikes flagged | 120 caught, 11 look-alikes flagged |
| Classic injections, of 60 | 55 caught, 2 of 56 benign flagged | 33 caught, 0 false flags |
d3, Decision Index 0.3.1 text table
vllm-sr/d3, opened October 11, 2026. One table. The card says the d3 row is an internal evaluation and the other rows are live board data dated 2026-10-10. The highlight's 65.45 and 56.97 are a different print. Vega's 56.5 is edition 0.2.1. We did not run the weights.
| Model | Index | Public |
|---|---|---|
| d3 | 64.7 | 65.5 |
| Perplexity Decider v1.1 | 62.8 | 62.3 |
| Jev | 60.1 | 58.0 |
| Decision 2.0 (27B) | 55.9 | 57.0 |
Mizan 4B, seven Persian tasks
xmanii/Mizan-4B, opened October 11, 2026. One mean over 3,000 records. The Q4 column is the same tasks. English held-out and the synthetic 91.2 are not in this table. We did not run the weights.
| Model | Mean accuracy |
|---|---|
| Mizan 4B | 78.7 |
| Mizan Q4 | 78.3 |
| Clef-flash 9B | 83.0 |
| Clef 27B | 83.4 |
| Jev | 83.7 |
| Laya 322M | 62.4 |
THX-01, four support-ticket sets
doofz/THX-01, opened October 11, 2026. One table, 2,843 tickets. Claude's row is a 200-ticket subset per set. Number lookup, where Jev is 96.6 and THX-01 is 93.4, is a different set. We did not call the API.
| Model | Average | ECE |
|---|---|---|
| THX-01 | 98.4 | 0.003 |
| Jev 1.13 | 97.4 | 0.007 |
| GPT-6-Luna | 97.7 | |
| Kev-4B | 92.8 | 0.202 |
PurpleMIST, translated African Typed Decisions
Olaverse card, opened October 10, 2026. One harness, translated set, 3,010 cases, 15,050 decisions. English typed-decisions and Decision Index 46.7 are other tests and are not in this table. We did not run the weights.
| Model | Size | Translated accuracy | KL |
|---|---|---|---|
| DeepSeek-V4-Pro, prompted | 1.6T | 0.669 | 1.125 |
| PurpleMIST-Flash-1.0 | 7.9B | 0.668 | 0.214 |
| Qwen3.8-Flash, prompted | undisclosed | 0.643 | 0.862 |
| OpenDecider-small | 4B | 0.477 | 0.365 |
| PurpleMIST-Mini-1.0 | 1.9B | 0.416 | 0.399 |
| laya-multilingual | 0.32B | 0.322 | 5.053 |
vegaml phishing screen, 800 emails
vegaml README, opened October 10, 2026. One paired run against live Jev 1.13.0. The README says the table is the rows where Vega leads. The 0.763 adapter score is a different test. Decision 2.0's Vega is a different checkpoint. We did not run the weights.
| Check | Vega 0.8B | Jev 1.13.0 |
|---|---|---|
| Accuracy, 800 emails | 75.4% | 61.9% |
| Caught, of 400 | 252 | 99 |
| Recall | 0.63 | 0.25 |
Author tables, September 29 and 30
Each row is one published table. The September 30 JevBench page numbers the addenda that the September 29 page left outside the ranked order. Matilda's 59.26 is the author's kit run. Jeeves leaves out the sealed judge tier. jgrep's 12% is a rounding of 0.119.
| Who | Set | Figure | Jev on that test | Caveat |
|---|---|---|---|---|
| JevBench v1.5.4 | 1,624 decisions, 106 ranked | Cygnet 73.7 · Winnow 73.2 · Plumb rank 5 at 71.6 | 72.1, official rank 3 | Same top three as September 29. Addenda are numbered and still tagged. Tie line is 69 of 77 pairs with a published comparison. |
| Matilda | Decision Index 0.2.1, self-run | 59.26, median 56.6 ms on the card | 57.91 on the September 28 board file | The card says the run is being submitted. The post's text says 56.8 ms. Knowledge and Reasoning is 43.52 against 51.40. |
| Jeeves | 111 public hard items, thinking | 0.865 | 0.730, Kev-published | Sealed judge tier left out. Transfer overall is 0.746 against Jev 0.800. Test overall is 0.889 against 0.857. |
| jgrep tests | 60 commits, five repos | recall 0.926, selection 0.119, $0.1112 | model field jev-latest | The reply's 12%, 90%+, and $0.11 round those cells. Not dzhng/jevgrep. The CLI does not echo the response model id. |
| Ollaya | README pair, item count unprinted | winnow:e4b 0.722, 89 ms for five questions | 0.738 | Port 11435. Independent of Ollama and of TypeSafe. JevBench's Winnow 73.2 is a different number. |
| Judge cascade v3 | 4,850 public judgments | 1.7 points behind GPT-6, $0.044 per 1,000 | JEV 1.13, median 0.15 s | Frozen cascade is +0.9 points on 1,610 pairs at 41% of the fee. Not the LangChain weather-agent repo. |
| JevAdvBench | Abstract, 812 questions | appended opinion flips 12.1% | jev-1.13.0, command 10.1% | Submitted September 25. We read the abstract, not the 19 tables. Not the 510-case InjecAgent paper. |
| OpenAI Decisions, October 6 guide | Public beta, gpt-6-luna | $0.10 per million input tokens | published list price $0.042 per million input | The guide lists no output charge. The September 29 extract still says 150 ms against 1.6 s and left the price unprinted. Choice and score answers include confidence. Not a TypeSafe route. |
| OpenAI Jev, October 6 | POST /v1/decisions, this desk | HTTP 200, 420 input tokens | September 30 TypeSafe call HTTP 200 | Processing header 354 ms. Desk clock 1,858 ms. Choice confidence 1.0. Empty POST HTTP 400. October 2 and October 4 were HTTP 403. The recap, opened October 6, still said limited preview. |
| Decisions and Jev, three texts | One call each, October 6 | cat 0.98 · urgency 0.99 · technical 0.99 | 0.98 · 0.98 · technical 0.74 | Stripe sentence from the TypeSafe quickstart. Frustration matched at 1.0. Leftover probability 0.01 on sales, 0.17 on billing. Clocks are one sample each. |
| Liquid d1, internal Decision Index 0.2.1 | Chart, five areas | d1 58.9 | Jev 57.9 | Arts +7.8, language +5.6, retrieval +5.3, tools -1.0, knowledge -8.0. Not a row we confirmed in the September 28 board file. |
| Open d1, d1-3B card | Decision Index 0.2.1, self-scored | 48.57 | no Jev row on that table | Winnow-12B on the same table is 50.02. Decider 35B-A3B is 47.11. d1-omni-600M is 15.95. Hosted d1's 58.9 and JevBench's 73.0 are other tests. |
| Drex 1.5, Nace chart | 231 public items, author's table | 199/231 · 86.2% | 201/231 · 87.0% | The September 30 post says JevBench SOTA. The chart prints Jev ahead, and a hard-tier tie at 82/111. October 1 Benchmark Heaven text has no Drex 1. |
| Rox rerank | Mean over 75 queries, relevance kept at 200 chunks | Jev-Score 96% | Production GPT-5 Mini 84% | Post also says 20x faster and 10x cheaper. Those multiples are not on the chart. |
| Jev Sentinel estimates | Demo page, per million actions | about $55 and 0.23 s | about $6,900 frontier judge | TechCrunch prints $2.94 versus $372 for one monitoring job. Those two figures are not on the demo page. |
| Mercury Decide | Inception post, JevBench v1.4 claim | no score printed | Jev 72.1 on the v1.5.4 page | Opened October 8, 2026, the API board ranks Mercury Decide third at 72.4, and the free URL returned the model page, priced Free. OpenRouter free URL returned Not Found. October 1 leaderboard text has no Mercury Decide. |
Author tables, September 28 and 29
Each row is one published table. Opened October 8, 2026, the API board ranks Jev 1.13.0 fourth at composite 71.5. The API board opened October 7, 2026 ranks Jev 1.13.0 third at composite 71.5. JevBench v1.5.0 was the official composite on the page opened September 29. 67.37 and 63.29 are the previous board. Violoop's Tetris gravity differs between the two models. Solar Decide's 87.0% does not print a numerator.
| Who | Set | Figure | Jev on that test | Caveat |
|---|---|---|---|---|
| JevBench API board, October 7 | v1.6.1, revision v1.7.12 | Sage 74.0 · d1 73.0 | 71.5, composite rank 3, capability 77.1 | Headline scores unchanged in the v1.7.12 note. 72.1 is the v1.5 composite. |
| JevBench v1.5.0 | 1,624 decisions, harmonic mean | Cygnet 73.7 · Winnow 73.2 | 72.1, official rank 3, capability 80.0 | Page calls the top pair a statistical tie. 89 of 103 ranked. Imajev would place sixth at 70.4, outside the ranked order. 63.29 is v1.4.2.2. |
| Violoop | One QuickBooks bill, 123 decisions | Opus 5.5: 978.9 s, $1.38 | 127.3 s, $0.0027, median 390 ms | 30 of 30 rows. List-rate model cost. Tetris gravity is 0.2 s against 0.4 s, so the piece count is a different clock. |
| Polylane | Five fixed-answer on-call steps | prior p90 4,752 ms, $0.76199 | 508 ms, $0.46369 per 1,000 | No accuracy and no call count. A later scatter does not match the task rows. |
| JevScout | Four printed MCP rows, five runs | comments 1 of 5 | comments 5 of 5, agent time -72% | Post says 7 tasks, -38% time, -26% cost. TypeSafe's token bill is separate. The note does not name tasks 5 to 7. |
| DTap | Tool-selector chart, no task count | direct ASR 70.1, benign 60.9 | self-gate 36.1 and 55.8 | Indirect injection 43.5 to 21.6. Leaderboard showed 0 agents. Not the 510-case InjecAgent paper. |
| Solar Decide | Public JevBench 231 | 87.0%, p50 0.14 s, p95 15.7 s | 86.1%, p50 0.30 s, p95 0.40 s | 87.0% of 231 is 200.97. Open-Jev's Jev cell on this split is 200/231, 86.6%. AskUp has no Jev row. |
| Glean rerank | 4,855 captured queries | Recall@6 40.2% without the tie-break | 44.3% with production order as the tie-break | About $0.00044, p50 0.195 s. About 37 of 41 candidates tied. Absolute recall is not the production ranker. |
Author tables, September 27 and 28
Each row is one published table. They are not one leaderboard. Imajev's 67.37 and Plumb's 65.84 are the v1.4.2 composite, the board before v1.5. Plumb's 89 of 111 is a different runner. The 400M encoder is not Palmer's Kev, and jevbench.app is not Florian's board.
| Who | Set | Figure | Jev on that test | Caveat |
|---|---|---|---|---|
| Imajev-4B | JevBench v1.4.2.2 composite | 67.37, rank 1 on that board | 63.29, rank 4 on that board | Capability view 66.3 against 64.7. DecisionBench ECE 0.024 to 0.069. Scenario checks 129/145 to 113/145 once the calibration file lowers confidence. The v1.5.6 page opened October 4 lists the same 70.4 as official rank 10. Image JevBench v0.1.5 lists 76.4 at rank 2. Opened October 8, 2026, Image JevBench v0.3.0 lists capability 68.0 at rank 1 and composite 63.7 at rank 2 of 52. |
| Plumb-4B | 231 public items, hard tier | 89 of 111, 0.802 | start checkpoint 82 of 111 | The v1.4.2.1 composite is 65.84. On v1.5 the addendum would place it fourth at 71.6, outside the ranked order. Hard ECE 0.094, worse than v4's 0.051. Dates and numbers stay 7 of 15. |
| skundu42/kev | 17,476 of its own rows | 79.96% | 73.54% on the same rows | Any positive-target candidate counts. PhishNChips 50.25% against a published 62.6%, 995 false flags. Not Palmer's Kev. No LICENSE file. |
| jevgrep | 10 tuned SWE-bench tasks | 8 of 10, combined bill -25.8% | same 8 of 10, saved $7.6221 | Jev list price included. Five tasks cost more. The September 26 post said 40%. Baselines were not rerun. |
| toolgate 0.15 | 10 matched pairs | 19 of 20, zero permissive allows | Jev scores the call | A classifier, not a sandbox. The 1,369-decision week is version 0.9.1, and the miss there was too many asks. |
| Deji | 100 grant posts, one day | 75 of 78 nos hidden | no first-pass accuracy printed | 5 of 5 yeses shown. The earlier key had 7 real yeses. No repository. We did not see the posts. |
| jevbench.app | 21 lead-qualification examples | 20 of 21 | the recorded answers | Miss is a yes at 0.51. Opening a test makes no new call. Not Florian's JevBench. |
| Christoph Görn | 8 prompts, local Kev-4B | 1 of 7 | OpenRouter arm 1 of 5 | Palmer's 4B, not the 400M encoder. A log of typesafe/jev-router names three general-purpose models, with 7 of 16 errors. |
Charts and papers, September 25 to 26
Each row is one published table. Deel's charts have no denominator. The attack paper is limited to jev-1.13.0. RLCD's 0.886 is a median AUROC, and the calibration line next to it is worse than a base-rate null. jev-code's ladder is one run per task. The Astra percentage is input tokens on one fixture, not the file-count drop.
| Who | Set | Figure | Jev on that test | Caveat |
|---|---|---|---|---|
| Deel / TypeSafe | Eight quality checks, no denominator | 6 of 8 at parity or better | messy metric questions -16.6 points | Expense +36 is against a rule-based matcher. Tweet says 50% to 86% and 70% to 97%. Chart prints the point gaps. |
| Wu and Lim | 510 InjecAgent cases, jev-1.13.0 | validated ASR 1.8% to 3.5% | the model under test | Score feedback. CHR at confidence 0.7 stays 0.2%. No repository in the paper. |
| RLCDAlignBench | 7,193 instances, 44 benchmarks | median AUROC 0.886 | Jev is the scorer | Median ECE 0.168 against a null of 0.074. AYi prints 0.31 s and 93%. Those are not in the tables we transcribed. |
| jev-code | Three coding tasks, one run each | 0 of 3 pass on 517d0f0 | Jev picks the productions | Hello world does pass: 3 turns, 16 requests. Dated September 18. Not run in CI. |
| astra-jev-harness | One two-file fixture, six checks | Astra input -27.0%, checks 6/6 | 2 Jev calls, 6,784 / 76 tokens | September 22 files fell 28 to 4 while Astra input fell 1.2%. One trial. Formula, not a Codex bill. |
Email routing
Three inboxes, three label sets. Vogel posted a sample size and no error rate. Mudholkar posted a named-company table with a Gemini baseline. Kumar posted agreement with whether a human kept the task.
| Who | Set | Jev | Against | Caveat |
|---|---|---|---|---|
| Vogel | 1,500 personal mails | n = 1,500 | none posted | No accuracy, no comparator. |
| Mudholkar / Bryo | 1,565 supplier mails, 10 categories | 96.4% | Gemini 3.5 Flash-Lite 97.5%; Gemini 3.8 Flash 98.5% | 364 synthetic mails. Attachments excluded. |
| Kumar | 800 inbox mails | 87.4% vs human keep | none posted | No lossless drop line. 96% on the confident band (69% of the set). |
Banking77
Same 77-way intent set, different comparators. Kumar ran 300 items against small generative models. Poly ran the full test against open zero-shot encoders and published ECE. NIRNAY's phase_b JSON is 0.8792 on 3,080 test rows. The card prints a Jev cell of 0.803 and also says that column was not measured in the repo. The October 3 post prints 0.8656 against a Jev figure of 0.64 and does not link the page that prints 0.64.
| Who | Slice | Jev | Against | Caveat |
|---|---|---|---|---|
| Kumar | 300 items, accuracy | 76.0% | gpt-5.4-mini 78.7%; gpt-5.6-luna 81.7% | Jev leads his three short-text sets and loses this one. |
| Poly | Held-out test, macro-F1 | 0.782 ± 0.009 | ModernBERT-large-zeroshot 0.712 ± 0.020 | ECE favors ModernBERT (0.081 vs 0.105). Thread only, no repo. |
| NIRNAY phase_b | 3,080 test rows, accuracy | 0.803 on the card | 0.8792, fine-tuned | JSON has no Jev field. The card says the Jev column was not measured in the repo. |
| NIRNAY post | phase_a 0.8656 | published 0.64, unlinked | 0.8656 | Laya's README prints a published Jev Banking77 of 0.870. The card's Julia-1 cell is 0.64 on 100 items. |
Local stand-ins
Jev's weights are not public. Each project scores itself on its own set, except JevBench, which puts several of them on one rubric. LocalJev does not put hosted Jev on the same 120 labels. Laya's Banking77 figure is the author's, against a published Jev number the README says was not measured in that repo. Palmer's 79.6% is his new-source split, not JevBench. Eikos's 82.9 is his public hard row. CLM's 31/38 is a verifier over sampled patches. v1.4.2's 64.13 is a harmonic mean with a sealed set. Jev's 63.29 on that formula is not the older geometric 75.4. Decision Index 0.2's 51.67 is a third number. jevos-1b's README alt text is 0.815 on 2,000 yes/no questions, against Jev at 0.927.
| Project | What | Their score | Jev on that test | Caveat |
|---|---|---|---|---|
| jevos-1b | 1B GGUF, 2,000 yes/no questions from unseen policies | 0.815 · short 54 ms | 0.927 · short 344 ms | README image alt. Laya on the same line is 0.489 and 104 ms. Choice and Score return 422. No LICENSE file in the root listing. |
| Nimble | 9B Qwen LoRA, 324 synthetic labels | 90.12% | 93.21% (jev-1.13.0) | Labels unreviewed by people. |
| Nimble, public suite | Earlier 9B checkpoint, 13 human-label subsets | 74.8% macro | 76.0% macro (jev-1.13.0) | 3,880 records. The September 24 checkpoint has no new row. |
| jeff | GLiFormer, 1,600 public items | AG News 75.5% | 90.5% | Copies POST /v1/systemone. Isolation of each noul is the default. |
| SemIf | Qwen3.5-4B logit readout, JevBench | 74.7 composite · 59.5% hard | Jev 75.4 · 74.1% hard | Closest open row on that suite. Author 3090 note is a different fixture. |
| LocalJev | Prompted JSON on oMLX, 120 gold labels | Qwen3.6-35B-A3B 76.7% short-input | not compared | Authors say the probabilities are not a logit read. |
| llamacpp-jev | Unmodified llama-server, Qwen3.5-2B, four questions on a 448 image | 526 ms median, 32/32 synthetic | not on this fixture | Raw label softmax. 64-way answers collapsed to option 16. Idle L4 was 236 ms. |
| kishida llama.cpp jev | Modified llama-server, 1,191 text questions, 2 to 8 options | Flash-Next 0.944 | guessing 0.283 | Mac 292 ms. 4060 Ti rows are a different machine. Not the 32/32 image check. |
| llama.cpp pull 29818 | Five decision-model GGUFs, one question, RTX PRO 6000 | 3 ms to 43 ms median | not on this fixture | Merged October 2. OpenJev reads images. The other four do not. |
| llama.cpp v0.6.0 | Release October 5. Clef on /v1/systemone. Nimble pull names Bespoke-Nimble-9B-v3 | Clef: no printed ms. v3 card: 56.88 | board Jev cell 57.91, run file not opened | 56.88 is Decision Index 0.2.1, CC BY-NC 4.0. Not the 324-row holdout. The October 6 note omits Nimble. |
| Verdict | 151M encoder, 337 TypeSafe public cases | 48.1% | 90.8% | Cited next to Nimble, not as its own story. |
| Laya | ModernBERT-sized encoder, JevBench plus author table | JevBench 70.1; Banking77 0.425 | Jev 75.3; published Jev Banking77 0.870 (72 labels) | Author Jev column was not measured in that repo. 77 vs 72 labels. |
| Kev | Qwen LoRA plus pointer head, author new-source plus JevBench | Kev-8B 79.6% new-source; kev 0.6B JevBench 66.7 | Jev 85.7% on Palmer's split; Jev 75.4 on JevBench | Author suite is decision-v7 / transfer-v4. JevBench rows are Qwen3 checkpoints. |
| Kev-4B, paired BFCL | Pinned 139fdd94, 1,253 function-selection cases, September 26 | 1,216/1,253 full JSON · 1,059/1,253 prose | 1,235/1,253 full JSON · 1,236/1,253 prose | Loopback HTTP versus hosted HTTPS. Not Palmer's 79.6% table. |
| djev | Hosted DiffusionGemma API, JevBench | 74.3 composite · 69.5% hard | Jev 75.4 · 74.1% hard | Announced $0.035/M, free preview. Native images. Calibration 65.4. |
| DiffusionGemma as Jev | One-step canvas read, eight hand sets | 198/201 | 191/201 | Author's chart. One read faster on 7 of 8 sets. Auto re-reads slower on all 8. Public 231 is 196 against 200. Routing joint is 32.2% against 61.4%. |
| SimpleJev Qwen3.8-27B | Next-token logit server, JevBench public demo | 67.3 composite · 75.0% hard | Jev 75.4 · 74.1% hard | Demo x2 latency adjustment. Cost estimated. Text HF server; site has vision demos. |
| Open-Jev 9B | Qwen3.5-9B LoRA plus decision head, 231 public JevBench tasks | 179/231 (77.49%) · hard 66/111 | Jev 200/231 (86.58%) · hard 81/111 | Public slice, not the 534-task composite. SemIf used the name OpenJev earlier. Astra is 231/231 on this slice. |
| Open-Jev 27B v1.1 | Qwen3.8-27B LoRA plus decision head, same 231 tasks | 197/231 (85.28%) · hard 80/111 | Jev 200/231 · hard 81/111 | September 23 card. No latency. Internal hard-label rates use another denominator. |
| Eikos-27B | Author harness, public JevBench hard tier | 82.9 · error at 0.90 is 2.4% of 7,140 | 73.0 · 5.9% error at 0.90 | No denominator on the hard cell. 64k probe has no Jev number. Post says eight batteries; README confidence row is six suites. |
| Eikos Chess | 388 visitor-started games, every legal move as an option | 42 wins · 53.9% · Elo +27 | 12 wins | Screenshot 22:54 UTC September 24. 334 draws. 95% interval +14 to +40. Answer time 3.5 s versus 288 ms. One finished 32-game round was 1 win, 31 draws, 0. |
| Eikos arena | Paper trading, cycle 626 of 864, 09:20 UTC September 26 | equity $9,653.19 · Brier 0.2716 | equity $10,051.22 · Brier 0.2500 | Same 72-hour run. Hit rate 50.2% versus 45.9%. Missed 0 versus 20. The September 25 file was cycle 302. |
| CLM-8B | Verifier over sampled patches, H100 | DeepSWE 31/38 · Terminal-Bench 2.1 87.6% of 30 | below pass@1 on that setup | Opus 5 and Fable 5 wrote the candidates. Not the May DeepSWE agent board. Speed there is 4.1 to 5.7 times Jev. |
| CLM, earlier BFCL column | 1,253 function-selection cases, reused September 26 | 76.86% description · 47.49% full JSON | 97.77% description · 98.56% full JSON | Historical CLM column, not a new CLM run. Jev numbers are the fresh paired run. |
| decider-4b v2 | JevBench v1.4.2 harmonic mean | 64.13 · Intelligence 49.4 · p50 0.02 s | 63.29 · Intelligence 53.1 · p50 0.65 s | Changelog September 25. Page shows 64.1 and 63.3. Public 83.5% against 86.6%. Sealed 34.7% against 36.7%. Cost is an estimate, about $0.020 against $0.040. The post says about 5 times faster. |
| Jev-Omni | JevBench v1.4.1, 534 public plus 308 sealed | 51.3 · public 88.7% · sealed 32.1% | 63.3 · public 86.6% · sealed 36.7% | Harmonic mean. The post says hard tier 75.0% against 74.1%. Not the v1.2 geometric 75.4. |
| Decision Index 0.2 | 40 public benchmarks, chance-corrected | AutoJev-27B 50.94 | Jev 51.67 | Kit test. Tie band is 0.25. A September 25 post says Drex is 51.73. That figure is not in the test we read. |
| Decision Index 0.2.1 | 38 benchmarks, board file September 28 | Rune 26B-A4B v3 57.44 | Jev 57.91 | 70 models plus Jev. No row named Drex. Nace's page prints Drex 1.5 at 58.28. The kit's test locks Jev at 57.89. |
| rev, Qwen3.8-27B | Frozen INT4, exact logprobs, public tiers | hard 0.784 · standard 0.986 | README's Jev hard cell 0.730 (also prints 0.721) | Same 48/72/111 headers. Not the leaderboard composite. 27B path includes two network hops. |
| rev, Qwen3.5-2B | 8-bit MLX on an M2 Pro, public tiers | hard 0.550 · standard 0.764 | same Jev cells as the 27B row | No weight training. 8-bit versus 4-bit is described as about 9 hard points. |
| spark-s1-4b-v6 | Qwen3.5-4B LoRA, public-tier proxy | Intelligence 83.1 · hard 0.595 | Kev dev partition, published Jev 0.857 vs 0.756 | Judge tier private. Not the official composite. NVFP4 hard cell is 0.622. |
| hotdog-27B | Binary LoRA, two sets | 1,603/1,631 holdout | 1,587/1,631 holdout; 62/74 vs 60/74 public binary | Holdout shares training sources. Public interval includes zero. Not the composite. |
| Rizzo Flow Q8_0 | Spark-X2.5-4B, SemIf's authored144 | 0.812 · held-out 0.793 | SemIf published 0.819 · 0.811 | Paired gap includes zero. 6 of 36 missing-evidence rows stay confident. SemIf's count is 1. Not JevBench. |
| Jevre 0.85B | Own 12,000-row table | test 0.9323 · held-out 0.8366 | not compared | ECE 0.0079 on test, 0.0944 on held-out sources. State shuffle falls to about 0.43. |
| jev_local Sarashina Q4 | JGLUE test, JNLI and JCommonsenseQA | 34% · 86% | not on this table | LFM on the same API is 17% and 69%. ModernBERT, trained on the train split, is 93% and 92%. |
JevBench public subset
Zefan Cai scored the 231 public JevBench tasks (72 original, 48 easy, 111 hard). The other 303 tasks are unavailable, so none of these rows is the 534-task composite. The September 21 rows ran Open-Jev on one H100 over loopback HTTP. Hosted models used HTTPS. GPT probabilities are verbalized. Candidate order differs on 119 of 139 Choice tasks. One observation per task. The 27B row is a September 23 card with a 16,384-token limit and no printed latency.
| Model | Correct / 231 | Hard | P50 | Caveat |
|---|---|---|---|---|
| Open-Jev 2B | 150 (64.94%) | 46/111 | 138 ms | Released Qwen3.5-2B LoRA. Prefix cache off. |
| Open-Jev 9B | 179 (77.49%) | 66/111 | 189 ms | Released Qwen3.5-9B LoRA. P95 839 ms. |
| Open-Jev 27B v1.1 | 197 (85.28%) | 80/111 | not printed | Qwen3.8-27B. Separate protocol. Three behind Jev, one behind on hard. |
| Jev 1.13.0 | 200 (86.58%) | 81/111 | 291 ms | One vector renormalized. About $0.038 per 1,000, estimated. |
| GPT-5.6 Luna | 206 (89.18%) | 89/111 | 954 ms | Reasoning none. About $0.179 per 1,000, estimated. |
| GPT-6 Astra | 231 (100%) | 111/111 | 2,206 ms | Reasoning low. About $8.82 per 1,000, estimated. |
Document classify and split
DocJev ran hosted Jev 1.13.0 and GPT-5.6 Luna on the same LiteParse page text. Forty authentic public-sector PDFs for classify, eight constructed packets for split. Decision times exclude OCR. Human annotation review was not performed.
| Task | Set | Jev | Luna | Caveat |
|---|---|---|---|---|
| Classify | 40 originals, eight per category | 40/40 · 138.6 ms median | 40/40 · 794.3 ms median | 5.73× median-time ratio. Convenience sample. |
| Split | 8 five-document packets, 116 pages | 7/8 exact · 209.6 ms median | 8/8 exact · 1,352.3 ms median | Jev extra cut on a Fed attachment. 6.45×. |
SQL row judgments
Five listings call a provider from SQL. Jevflake stores answers in Snowflake through dbt. duckdb-jev batches inside DuckDB and published a live Choice throughput table. MotherDuck's prompt_jev() runs inside the warehouse on paid plans and published topic accuracy. Zachi's pg-jev batches Postgres rows and prints a 2,000-row timing. Mejias's pg-jev is a different PostgreSQL extension and prints no scored set. All five send row text out of the database.
| Who | Host | Published figure | Caveat |
|---|---|---|---|
| Kranz / Jevflake | Snowflake SQL via dbt | v0.1, one live account, ten sample tickets | Packing several rows is experimental. Extra columns changed probabilities. |
| Prasanth / duckdb-jev | Native DuckDB C++ | 2,049 rows in 0.887 s; cache 28.6 ms | Twelve-template corpus. Throughput, not accuracy. colliber/duckdb-jev is a separate MIT extension. |
| MotherDuck / prompt_jev | Hosted SQL, paid plans | 100k AG News train rows, 89%, 40s, $0.50 | Training split, seed 43. Terra on the same sample is 88%, 31m 59s, $37.58. Nulls unscored. |
| Zachi / pg-jev | PostgreSQL extension, plpython3u | 2,000 rows, about 3.5 s, about $0.012; cache about 50 ms | Author timing from Europe. Ground truth 100% at batch 20, 92% to 98% at 40. pgjev.com still says batch 40 and concurrency 6. |
| Mejias / pg-jev | PostgreSQL extension, plpython3u | no scored set in the README | MIT. A different repository. Returns jsonb. One call per row. Rollback does not undo a request already sent. |
Coding-agent tool routing
jev-gateway's README cites 120 sessions on a chess rules engine: six models, fix-bugs and add-a-feature, five runs with routing and five without, no MCP servers or plugins. Cells are medians versus the same model with routing off: output tokens, input tokens, then time. Claude Code rows are hint mode. Five runs per cell is the sample the README flags as small.
| Model | Client | Bug-fix | Add a feature | Caveat |
|---|---|---|---|---|
| GPT-6 Astra | Codex | -57% / -7% / -39% | 0% / +2% / +8% | The tweet's 57% is this bug-fix output cell, shared with Sol. |
| GPT-5.6 Sol | Codex | -57% / -40% / -36% | -9% / -39% / -16% | Both tasks used fewer tokens with routing on. |
| GPT-5.6 Luna | Codex | -12% / -10% / +10% | -14% / -51% / -14% | Feature task solved 3 of 5 with routing, 5 of 5 without. |
| Fable 5.1 | Claude Code | -13% / -19% / +6% | -24% / -27% / -26% | Hint mode. The model can ignore the suggestion. |
| Opus 5 | Claude Code | -7% / -22% / +2% | +22% / +61% / +83% | Feature task used more tokens and more time. |
| Sonnet 5 | Claude Code | -41% / -48% / -25% | +9% / +16% / +37% | Bug-fix got cheaper. The feature task did not. |
Site search ranking
McLaren indexed TypeSafe's documentation and ran the same labelled queries through jevsearch and several keyword libraries. Jev re-ranks a 20-hit keyword pool. A page that never reaches the pool cannot win.
| Library | Hit@1 | Hit@3 | Median | Caveat |
|---|---|---|---|---|
| jev-search | 83% | 83% | 278 ms uncached | 41 labelled queries. About $0.26 per thousand searches. |
| Keyword pass / Lunr | 41% | 59% | 8.7 ms / 0.7 ms | The keyword pass tied Lunr. Jev converts that 20-hit recall. |
| Pagefind | 20% | 29% | 2.6 ms | Returned nothing for 12 of 41 queries. |
Pairwise ranking
jsort fits a scale from pairwise Noul judgments. The CommonLit check uses teacher Bradley-Terry scores as the reference. The Fed check uses the funds-rate move, not a human ranking. Joe Weisenthal's 4,005-speech post is a separate count from the 95-statement script.
| Check | Set | jsort | Other measure | Caveat |
|---|---|---|---|---|
| CommonLit easiness | 300 excerpts, -k 10 | Pearson r 0.824 · $0.046 | One Noul r 0.754 · SMOG 0.661 | Teacher reliability about 0.78, ceiling near 0.88. Run 2026-09-19. |
| FOMC hawkishness | 95 opening statements | Same-day rate move +0.46 | Next 180 days +0.37 | Top six are the 2022 50- and 75-point hikes. 4,005 is Joe's later post. |
Judging other output
Same job, different oracles and sample sizes. LangChain froze five weather traces and used a human label. Lemkin compared Jev with Sonnet on Connect pairs and posted a false-admission count. Every asked Jev 21 writing questions and did not put an LLM judge on the same 777 cells. JevBench's hard tier is a different rubric, written by Opus 5 and GPT-5.6 Sol. RZ recorded 100 synthetic security spans five times each.
| Who | Set | Jev | Against | Caveat |
|---|---|---|---|---|
| LangChain | 5 traces × 100, does_pass | 500/500 vs human | Terra 99.8%; Luna 96.4%; Sonnet 4.6 80.0% | One reviewer. Quality variance is the other claim. |
| Lemkin / SaaStr | 600 judgments | 70.5% vs third-model referee | Sonnet 77.5%; 32% vs 13.5% false admissions | Thread only. No prompts, no repo. |
| Every | 37 docs × 21 questions | 777 answers in <0.7 s | later 12-passage check vs Fable 5.1, 6 of 7 defects | No LLM judge on the 777-cell grid. |
| JevBench hard tier | 220 frozen items | 74.1% | Luna 94.5%; SemIf 59.5% | Composite still ranks Jev above Luna on cost. |
| RZ / security triage | 100 synthetic cases × 5 | 65.3% balanced accuracy · 0.57 s | Terra 75.1% · 10.23 s; Opus 91.2% · 13.83 s | Labels by one author. Safe-class recall 45.5%. CLI vs API latency. |
Legal-set loops
The host enumerates actions the runtime can take. Jev picks among those options. Illegal moves stay out because they were never listed. These are demos, not a shared computer-use eval.
| Demo | Set | Figure | Caveat |
|---|---|---|---|
| Browser Use | Zurich to London Google Flights | 7.1 s · $0.0039 | Repo notes; clip at 1x. A desk loop on September 30 never left the start page. The story has that run. |
| Stagehand / jev-browser | Computer use from an a11y tree | $0.001, near-instant (Jeong) | Author post, not a Browserbase lab table. A three-site desk loop is on the story. |
| Tetris | Same 200-piece sequence, legal landings | 9,200 pts · ~300 ms · 0 illegal | One run. Haiku 8,900 / 1.52 s; Gemini Flash-Lite 9,000 / 1.13 s. |
| JevPilot | Sampled steering paths in Three.js | up to 4 requests/s; $1.24 / 15,000 | A game. Geometry stays in the browser. A headless sim on September 30 is on the story. |
| macOS Loop | Six native GUI tasks | 6/6; Finder 9 files in 7.39 s | Small functional sample. An earlier Calculator run failed. |
| Jevis | Flutter integration_test | no published pass rate | Typed strings come from the test author. |
Context filters
Tran argued for scoring tool output and dropping it. AstroHan kept the agent and the model fixed and reported whether the task still passed. One run per arm.
| Who | Set | With Jev | Without | Caveat |
|---|---|---|---|---|
| Tran / fast-jev-compaction | Keep-or-drop tool calls | plugin and library | summarization (the default it replaces) | No published pass-rate table in that repo. |
| AstroHan | FrontierHarness 30, deepseek-flash | 25/30 · meriyah 49/49 | 22/30 · meriyah 0/49 | Two of 30 the filter lost. 126m vs 123m input tokens. Thread only. |
Router tier on a labeled set
AgentOS compared an opt-in Jev strategy with its default on-device MiniLM classifier. Same 121 turns, gold labels R0 to R3, classifier only, two repeat runs identical. Blank cells are blank on the chart. A same-day follow-up says it sharpens the R2 and R3 wording. This is not a JevBench score.
| Metric | Local MiniLM | Jev | Caveat |
|---|---|---|---|
| Accuracy | 0.355 | 0.901 | 121 turns. Vietnamese, Chinese, English. |
| Under-routing | 0.413 | 0.066 | Gold is the project's own R0 to R3 labels. |
| Over-routing | 0.231 | 0.033 | Jev is opt-in. Turn text leaves the machine. |
| English / Vietnamese / Chinese | 0.513 / 0.312 / 0.250 | 0.946 / 0.938 / 0.806 | Chinese is the low Jev slice on this chart. |
| Cost and time | $0, on device | about $0.005, about 0.9 s per call | 121 cases. Macro-F1 0.894 and ECE 0.062 are Jev-only on the chart. |
Laya, two machines
Same weight families, different runtimes. The T4 times are from the Laya README cited in the earlier story. The M3 Max times are from the laya-mlx README. Neither row is a score against hosted Jev. laya-mlx also says the selected answer matched upstream on 63 of 63 validation questions.
| Checkpoint | Author T4 | laya-mlx M3 Max | Caveat |
|---|---|---|---|
| English, 421M | 39.5 ms, one question | 13.42 ms P50, 13.92 ms P95 | Different machines. MLX port, FP16, load excluded. |
| Multilingual, 322M | 32.8 ms, one question | 7.39 ms P50, 7.79 ms P95 | Batched multilingual on the T4 note was 7.2 ms per question at 10 questions. |
Picking a cube turn
maxli asked Jev for the best of 18 face turns, forty times, then removed the experiment. Kociemba in code is the solver that ships. A random walk is the other number in the same README note.
| Who picks | Distance | Time | Caveat |
|---|---|---|---|
| Kociemba, in code | at most 22 moves | about 20 ms | cubejs wrapper. This is the solve button. |
| Jev, best of 18 | 22 to 22 | forty calls | Experiment removed. The coach asks whether to speak instead. |
| Random walk | 22 to 21 | same note | One comparison the README prints beside the Jev run. |
Same Flights task
Zurich to London, one way, one adult, economy, stop when results are visible. The Jev column is jev-ultrafast's published measurement. dejevu did not rerun it. dejevu's own runs are September 22 on an M3 Pro, and the Sunday it searches is October 18 2026 because September 20 has passed.
| Agent | Verified | Time | Calls and tokens | Caveat |
|---|---|---|---|---|
| jev-ultrafast, Jev 1.13 + Mercury 2.5 | 3 of 3, published | 7.09 s | 17 calls, 84,650 in, about $0.0036 | Other machine. The earlier clip on this desk said about 7 s and $0.0039. |
| dejevu, llama-3.3-70b on Groq | 3 of 3 | 5.63 s | 10 calls, 15,071 in, $0.0092 | Faster here, and about 2.5 times Jev's list price on this route. |
| dejevu, same Llama on DeepInfra | 2 of 2 | 37.5 s | $0.0020 | Cheaper than the published Jev cell, and much slower the day they measured. |
Typed decisions
meraGPT's September 22 blog, one run, zero-shot for Decider 1 and for the Jev row. The fitted encoders were trained on each workflow's labels. Reference labels are the average of three teacher samples. The blog says 400 cases and 2,000 decisions. The dataset card, read September 30, says that count is the test split. The Hugging Face viewer lists 1,600 rows across four subsets, train and test together. We did not recompute the table.
| Model | Accuracy | KL | Brier | Caveat |
|---|---|---|---|---|
| Prior, ignores the input | 0.470 | 0.347 | 0.189 | Floor on the blog's table. |
| ModernBERT-base, fitted | 0.646 | 0.223 | 0.119 | Trained per question. Decider 1 did not get that. |
| Jev 1.13.0 | 0.727 | 1.442 | 0.148 | Choice gap vs Decider 1 is 0.013. The KL gap is the wide one. |
| Decider 1 (sd-1) | 0.768 | 0.096 | 0.052 | Hosted. 4,096 token cap. At most ten choice labels. $0.03 / 1M input. |
Banking-feed categories
Peter's September 22 post. 612 sanitized rows, described as a harder subset of about 33,000. This is not Banking77. Gemini costs are estimates and exclude thinking tokens. No category list was printed.
| Model | Accuracy | Time | Cost | Caveat |
|---|---|---|---|---|
| Jev 1.13 | 36.1% | 6 s | $0.023 | Closest to Gemini 2.5 on accuracy. Fastest of the three. |
| Gemini 2.5 Flash | 37.3% | 95 s | about $0.088 | Cost is an estimate from the text-token budget. |
| Gemini 3.7 Flash | 42.5% | 78 s | about $0.180 | Highest accuracy on this private slice. |
Hold'em clocks
Two published timings for Jinzhengxu/poker-table, plus the README's language-model baselines on the same 46 spots. The live pill and the spot bank are not the same measurement. The bb/100 table elsewhere in that README is a rule-policy ablation.
| Clock | Time | What was measured | Caveat |
|---|---|---|---|
| Live pill, September 22 post | 545 ms average, $0.002 a hand | The table the post links | Opened September 30. The table was waiting, and Jev was not configured. 545 and 0.002 are not on the page. |
| Spot bank, Jev | about 1 s median | 46 spots, 3 repeats, OpenRouter | No accuracy percentage. Three passes under a cent. |
| Spot bank, single-shot LLM | 6 s | Same spots, reasoning model | README says Jev's range read matched it. |
| Spot bank, tool-loop agent | 35 s | Same spots | The path Jev replaced in front of the action clock. |
Public easy, standard, and hard
Two September 22 READMEs score the public JevBench tiers and print the same headers: easy 48, standard 72, hard 111. They are separate runs. rev's Jev column was measured in that harness. spark does not print a Jev cell on this split and says the proxy is not the official composite. Open-Jev's hard count, 81/111, is 0.730, which matches the 0.730 rev prints, and rev also prints 0.721 for a same-day measurement.
| Model | Easy | Standard | Hard | Caveat |
|---|---|---|---|---|
| rev, Qwen3.8-27B exact | 1.000 | 0.986 | 0.784 | INT4, 2x L40S, 2026-09-22. 32 samples match. Not on the leaderboard. |
| Jev 1.13.0, rev's table | 1.000 | 0.986 | 0.730 | Hard cell also prints 0.721 measured the same day. |
| rev, Qwen3.5-2B 8-bit | 1.000 | 0.764 | 0.550 | M2 Pro, in process. No training. |
| spark-s1-4b-v6 bf16 | 1.000 | 1.000 | 0.595 | Proxy Intelligence 83.1. Judge tier private. |
| spark-s1-4b-v6 NVFP4 | 1.000 | 0.972 | 0.622 | Own-split accuracy for this copy was not remeasured. |
Three Japanese sets
t.toda's September 22 image. The fourth column is the mean of the three percentages. Item counts, the prompt, and a repository are not on the post.
| Model | JCommonsenseQA | JMMLU | JamC-QA | Mean |
|---|---|---|---|---|
| Jev | 96.0% | 90.2% | 56.0% | 80.8% |
| Bonsai 2, 1-token logit | 93.8% | 66.8% | 41.2% | 67.2% |
| GPT-5.6 Luna | 97.2% | 82.5% | 76.5% | 85.4% |
Probability words
Mike Taylor's September 22 chart. Twenty-five phrases, human answers from D. Hails (2026), n=99. Jev 1.13.0 answered one Choice per phrase, 0 to 100 in steps of 5. Both series are smoothed. We read the image.
| Series | What | Printed result | Caveat |
|---|---|---|---|
| 99 people | Where each phrase sits | ridges on the chart | Footer cites Hails 2026. Opened September 30. Probable is 70, rounded to the nearest 5. The essay has no Jev column. |
| Jev 1.13.0 | One Choice per phrase | rank agreement 0.99, median gap 3 points | Gap described as about the same as between two human surveys. No per-phrase mean in the caption. |
Ámbar Reflex, same questions
Abel Coronado's September 23 README. Three local models, no generated text, one original run-and-gun. Offline accuracy is 554 labelled decisions with no clock. Realtime progress is a separate column. The questions were tuned on the 4B.
| Model | Offline accuracy | Realtime progress | Latency | Runs |
|---|---|---|---|---|
| Qwen3.5-4B Q8_0 | 77.8% | 24.9% | 408 ms | Realtime N=5. Questions tuned on this model. |
| Gemma 4 E4B Q6_K | 81.6% | 14.4% | 312 ms | Jumps on 74% of decisions. |
| Qwen3.5-9B Q5_K_M | 82.7% | 17.5% | 591 ms | Highest offline accuracy. Less progress than the 4B. |
| Same 4B, writing JSON | not in the 554 | 10.3% | 1,041 ms | Realtime N=3. Random agent is 4.6% at N=5. |
One screenshot, two prices
awlevin's README, one decision, the same screenshot and goal. The Opus column reads the image. The Jev column reads OCR text, and dates are annotated in code before the call. No task success rate is printed.
| Model | Input tokens | Cost | Model latency | End to end |
|---|---|---|---|---|
| Jev | 4,882 | $0.0002 | 0.13 to 0.38 s | about 1.5 s |
| Claude Opus 5, bare screenshot | 4,785 | $0.032 | 5.2 s | about 5.5 s |
JGLUE test, three local backends
Argos1111/jev_local README. Same API, JGLUE test. ModernBERT was trained on the JGLUE train split. The page says the rows are not a fair ranking. There is no hosted Jev column. JNLI for Sarashina is still under a majority baseline, on the author's Sarashina note.
| Backend | JNLI | JCommonsenseQA | Caveat |
|---|---|---|---|
| LFM2.5 | 17% | 69% | First-token logprob. Default backend. |
| Sarashina2.2 Vision 3B Q4 | 34% | 86% | Choice cap 26. Post latency about 500 ms on a Radeon AI Pro R9700. |
| ModernBERT-Ja 310M | 93% | 92% | Trained on the JGLUE train split. |
Four judges, classical sentences
hw725/dansa-research, September 23. 28,074 Korean translations. Chat models scored twenty sentences per prompt. Jev scored one sentence per call, O at 0.5 or higher. The gap is target O rate minus control O rate, in points. There is no gold label. Adding Jev leaves the book-stratified odds ratios at 8.64, 2.17, and 1.78.
| Judge | Section 1 | Section 2 | Section 3 | Caveat |
|---|---|---|---|---|
| GPT-5-mini | +46.5 | +3.8 | +36.8 | Text answers, parsed to O or X. |
| Gemini 2.5 Flash | +39.3 | +1.0 | +22.0 | Section 2 barely separates the endings. |
| Claude Sonnet 4.6 | +47.0 | +18.5 | +31.8 | Largest section 2 gap among the chat models. |
| Jev, jev-latest | +38.3 | +11.3 | +36.8 | 28,089 calls, about $0.54. 342 items sit on 0.5. |
100 English items
码农暖爸, September 23. The author's items, not Banking77 and not JevBench. Laya is the English checkpoint. The post says the set is not a full benchmark. The item list is a reply image.
| Model | Correct | Wrong | Caveat |
|---|---|---|---|
| Laya, English | 92 | 8 | Local checkpoint. The post does not print which eight. |
| Jev | 100 | 0 | Cloud call. No repository. We did not rerun it. |
Local files after a word match
assadiandre/jev-search, September 23. nDCG@10 is the only row printed as BM25, then JEV. The top-5 and top-100 cells are one number per set. kylemclaren/jevsearch is a different task: Hit@1 on 41 TypeSafe-docs queries.
| Set | Queries | BM25 nDCG@10 | JEV nDCG@10 | Caveat |
|---|---|---|---|---|
| NFCorpus | 323 | 0.322 | 0.377 | Top-5 relevant 70.9%. Top-100 documents 25.5%. 0.084¢. |
| SciFact | 300 | 0.687 | 0.745 | Top-5 relevant 80.3%. Top-100 documents 92.9%. 0.130¢. |
| TypeSafe docs, jevsearch | 41 | keyword Hit@1 41% | Hit@1 83% | Different metric and corpus. $0.26 per 1,000 queries. |
Bag of decisions, one runner
Doug Turnbull, October 8, 2026. Six rows from one CSV at commit 8734fde. Direct and bag share title^9.4 and description^4. The BM25 row uses title 9.3 and description 4.1. Means are NDCG@10 as cheat-at-search defines it: gain (2^grade - 1) / rank, ideal a perfect top 10. The two WANDS Jev medians are one float. We did not rerun the searches.
| Dataset | BM25 mean | One Noul | GPT-5 questions | Median, one Noul and the bag |
|---|---|---|---|---|
| WANDS | 0.540755 | 0.574262 | 0.609814 | 0.560945 for both |
| ESCI | 0.289473 | 0.313621 | 0.360079 | 0.204850 and 0.341417 |
A typed gate after retrieval
Each row is the author's own suite. CLERC is one gold passage per query. LoCoMo recall@1 is evidence turns. NFCorpus nDCG@10 here is aifabrice's BM25 top 30, not andre's top 128. SciFact is Lyzr's top 20 dense chunks. jevrag's HotpotQA row is exact match, not a rank. We did not rerun them.
| Suite | Before | After Jev | Setup |
|---|---|---|---|
| CLERC, 40 queries | top-1 5% | top-1 18% | jev-1.12, BM25 shortlist 30, 1,200 calls, $0.0645. Top-10 38% to 62%. |
| LoCoMo, 30 candidates | recall@1 0.800 | recall@1 0.950 | Hindsight listwise Choice. 0.12 s to 0.027 s. Pruning off. |
| NFCorpus, aifabrice | nDCG@10 0.306 | 0.353, best 0.451 | BM25 top 30, then Jev. Best row adds a planner. Passage Gate scored lower. |
| SciFact, Lyzr | nDCG@10 0.776 | 0.836 | jev-1.13.0, one call, top 20. bge-reranker-v2-m3 is 0.774. |
| HotpotQA, jevrag | EM 0.4057 | EM 0.4129 | n=700. Rounds 2,100 to 1,291. McNemar p = 0.36. ECE 0.3322. |
42 claims, five runs
TheWayWithin/jev-bench, September 22 runs, September 23 page. Means on the hard 18. Ranges are in the story. One claim is 5.6 points. Majority-of-five McNemar against GPT-5.4 is p = 1.000. Florian's JevBench is the 534-task composite.
| Model | Hard 18 | Easy 24 | All 42 | Caveat |
|---|---|---|---|---|
| Jev, jev-latest | 73.3% | 91.7% | 83.8% | Range 72.2 to 77.8 on the hard 18. $0.000025 a claim. |
| GPT-5.4 | 70.0% | 100% | 87.1% | Range 66.7 to 72.2. $0.001289 a claim. |
| Claude Sonnet 5 | 66.7% | 100% | 85.7% | Range 61.1 to 72.2. $0.002259 a claim. |
| Gemini 3.1 Pro | 63.3% | 100% | 84.3% | Range 55.6 to 66.7. $0.004940 a claim. |
One needle, Jev on or off
elalber2000/jev-rlm, seed 42, one generated file of 100,000 lines. Both arms returned 2867825. The dollar figures are the post's estimate. results.json has no charge field.
| Arm | Time | Calls | Answer | Caveat |
|---|---|---|---|---|
| Jev on | 1,081 s | 82 Jev, 1 recursive | 2867825 | typesafe/jev-1.13. Post estimates $0.089. |
| Jev off | 1,480 s | 78 recursive | 2867825 | Same root model. Post estimates $0.155. |