Updated
Desk
Compare
Similar tests already in the stories, placed next to each other. The sets, labels, and comparators differ, so the rows are not a ranking. We did not rerun any of them. One-row write-ups live on evals. Failure modes live on limits.
Email routing
Three inboxes, three label sets. Vogel posted a sample size and no error rate. Mudholkar posted a named-company table with a Gemini baseline. Kumar posted agreement with whether a human kept the task.
| Who | Set | Jev | Against | Caveat |
|---|---|---|---|---|
| Vogel | 1,500 personal mails | n = 1,500 | none posted | No accuracy, no comparator. |
| Mudholkar / Bryo | 1,565 supplier mails, 10 categories | 96.4% | Gemini 3.5 Flash-Lite 97.5%; Gemini 3.8 Flash 98.5% | 364 synthetic mails. Attachments excluded. |
| Kumar | 800 inbox mails | 87.4% vs human keep | none posted | No lossless drop line. 96% on the confident band (69% of the set). |
Banking77
Same 77-way intent set, different comparators. Kumar ran 300 items against small generative models. Poly ran the full test against open zero-shot encoders and published ECE.
| Who | Slice | Jev | Against | Caveat |
|---|---|---|---|---|
| Kumar | 300 items, accuracy | 76.0% | gpt-5.4-mini 78.7%; gpt-5.6-luna 81.7% | Jev leads his three short-text sets and loses this one. |
| Poly | Held-out test, macro-F1 | 0.782 ± 0.009 | ModernBERT-large-zeroshot 0.712 ± 0.020 | ECE favors ModernBERT (0.081 vs 0.105). Thread only, no repo. |
Local stand-ins
Jev's weights are not public. Each project scores itself on its own set, except JevBench, which puts several of them on one rubric. LocalJev does not put hosted Jev on the same 120 labels. Laya's Banking77 figure is the author's, against a published Jev number the README says was not measured in that repo. Palmer's 79.6% is his new-source split, not JevBench.
| Project | What | Their score | Jev on that test | Caveat |
|---|---|---|---|---|
| Nimble | 9B Qwen LoRA, 324 synthetic labels | 90.12% | 93.21% (jev-1.13.0) | Labels unreviewed by people. |
| jeff | GLiFormer, 1,600 public items | AG News 75.5% | 90.5% | Copies POST /v1/systemone. Isolation of each noul is the default. |
| SemIf | Qwen3.5-4B logit readout, JevBench | 74.7 composite · 59.5% hard | Jev 75.4 · 74.1% hard | Closest open row on that suite. Author 3090 note is a different fixture. |
| LocalJev | Prompted JSON on oMLX, 120 gold labels | Qwen3.6-35B-A3B 76.7% short-input | not compared | Authors say the probabilities are not a logit read. |
| llamacpp-jev | Unmodified llama-server, Qwen3.5-2B, four questions on a 448 image | 526 ms median, 32/32 synthetic | not on this fixture | Raw label softmax. 64-way answers collapsed to option 16. Idle L4 was 236 ms. |
| Verdict | 151M encoder, 337 TypeSafe public cases | 48.1% | 90.8% | Cited next to Nimble, not as its own story. |
| Laya | ModernBERT-sized encoder, JevBench plus author table | JevBench 70.1; Banking77 0.425 | Jev 75.3; published Jev Banking77 0.870 (72 labels) | Author Jev column was not measured in that repo. 77 vs 72 labels. |
| Kev | Qwen LoRA plus pointer head, author new-source plus JevBench | Kev-8B 79.6% new-source; kev 0.6B JevBench 66.7 | Jev 85.7% on Palmer's split; Jev 75.4 on JevBench | Author suite is decision-v7 / transfer-v4. JevBench rows are Qwen3 checkpoints. |
| djev | Hosted DiffusionGemma API, JevBench | 74.3 composite · 69.5% hard | Jev 75.4 · 74.1% hard | Announced $0.035/M, free preview. Native images. Calibration 65.4. |
| SimpleJev Qwen3.8-27B | Next-token logit server, JevBench public demo | 67.3 composite · 75.0% hard | Jev 75.4 · 74.1% hard | Demo x2 latency adjustment. Cost estimated. Text HF server; site has vision demos. |
| Open-Jev 9B | Qwen3.5-9B LoRA plus decision head, 231 public JevBench tasks | 179/231 (77.49%) · hard 66/111 | Jev 200/231 (86.58%) · hard 81/111 | Public slice, not the 534-task composite. SemIf used the name OpenJev earlier. Astra is 231/231 on this slice. |
JevBench public subset
Zefan Cai scored five systems on the 231 public JevBench tasks (72 original, 48 easy, 111 hard). The other 303 tasks are unavailable, so none of these rows is the 534-task composite. Open-Jev ran on one H100 over loopback HTTP. Hosted models used HTTPS. GPT probabilities are verbalized. Candidate order differs on 119 of 139 Choice tasks. One observation per task.
| Model | Correct / 231 | Hard | P50 | Caveat |
|---|---|---|---|---|
| Open-Jev 2B | 150 (64.94%) | 46/111 | 138 ms | Released Qwen3.5-2B LoRA. Prefix cache off. |
| Open-Jev 9B | 179 (77.49%) | 66/111 | 189 ms | Released Qwen3.5-9B LoRA. P95 839 ms. |
| Jev 1.13.0 | 200 (86.58%) | 81/111 | 291 ms | One vector renormalized. About $0.038 per 1,000, estimated. |
| GPT-5.6 Luna | 206 (89.18%) | 89/111 | 954 ms | Reasoning none. About $0.179 per 1,000, estimated. |
| GPT-6 Astra | 231 (100%) | 111/111 | 2,206 ms | Reasoning low. About $8.82 per 1,000, estimated. |
Document classify and split
DocJev ran hosted Jev 1.13.0 and GPT-5.6 Luna on the same LiteParse page text. Forty authentic public-sector PDFs for classify, eight constructed packets for split. Decision times exclude OCR. Human annotation review was not performed.
| Task | Set | Jev | Luna | Caveat |
|---|---|---|---|---|
| Classify | 40 originals, eight per category | 40/40 · 138.6 ms median | 40/40 · 794.3 ms median | 5.73× median-time ratio. Convenience sample. |
| Split | 8 five-document packets, 116 pages | 7/8 exact · 209.6 ms median | 8/8 exact · 1,352.3 ms median | Jev extra cut on a Fed attachment. 6.45×. |
SQL row judgments
Four listings call a provider from SQL. Jevflake stores answers in Snowflake through dbt. duckdb-jev batches inside DuckDB and published a live Choice throughput table. MotherDuck's prompt_jev() runs inside the warehouse on paid plans and published topic accuracy. pg-jev is a PostgreSQL extension and prints no scored set. All four send row text out of the database.
| Who | Host | Published figure | Caveat |
|---|---|---|---|
| Kranz / Jevflake | Snowflake SQL via dbt | v0.1, one live account, ten sample tickets | Packing several rows is experimental. Extra columns changed probabilities. |
| Prasanth / duckdb-jev | Native DuckDB C++ | 2,049 rows in 0.887 s; cache 28.6 ms | Twelve-template corpus. Throughput, not accuracy. colliber/duckdb-jev is a separate MIT extension. |
| MotherDuck / prompt_jev | Hosted SQL, paid plans | 100k AG News train rows, 89%, 40s, $0.50 | Training split, seed 43. Terra on the same sample is 88%, 31m 59s, $37.58. Nulls unscored. |
| Mejias / pg-jev | PostgreSQL extension, plpython3u | no scored set in the README | MIT. Returns jsonb. One call per row. Rollback does not undo a request already sent. |
Coding-agent tool routing
jev-gateway's README cites 120 sessions on a chess rules engine: six models, fix-bugs and add-a-feature, five runs with routing and five without, no MCP servers or plugins. Cells are medians versus the same model with routing off: output tokens, input tokens, then time. Claude Code rows are hint mode. Five runs per cell is the sample the README flags as small.
| Model | Client | Bug-fix | Add a feature | Caveat |
|---|---|---|---|---|
| GPT-6 Astra | Codex | -57% / -7% / -39% | 0% / +2% / +8% | The tweet's 57% is this bug-fix output cell, shared with Sol. |
| GPT-5.6 Sol | Codex | -57% / -40% / -36% | -9% / -39% / -16% | Both tasks used fewer tokens with routing on. |
| GPT-5.6 Luna | Codex | -12% / -10% / +10% | -14% / -51% / -14% | Feature task solved 3 of 5 with routing, 5 of 5 without. |
| Fable 5.1 | Claude Code | -13% / -19% / +6% | -24% / -27% / -26% | Hint mode. The model can ignore the suggestion. |
| Opus 5 | Claude Code | -7% / -22% / +2% | +22% / +61% / +83% | Feature task used more tokens and more time. |
| Sonnet 5 | Claude Code | -41% / -48% / -25% | +9% / +16% / +37% | Bug-fix got cheaper. The feature task did not. |
Site search ranking
McLaren indexed TypeSafe's documentation and ran the same labelled queries through jevsearch and several keyword libraries. Jev re-ranks a 20-hit keyword pool. A page that never reaches the pool cannot win.
| Library | Hit@1 | Hit@3 | Median | Caveat |
|---|---|---|---|---|
| jev-search | 83% | 83% | 278 ms uncached | 41 labelled queries. About $0.26 per thousand searches. |
| Keyword pass / Lunr | 41% | 59% | 8.7 ms / 0.7 ms | The keyword pass tied Lunr. Jev converts that 20-hit recall. |
| Pagefind | 20% | 29% | 2.6 ms | Returned nothing for 12 of 41 queries. |
Pairwise ranking
jsort fits a scale from pairwise Noul judgments. The CommonLit check uses teacher Bradley-Terry scores as the reference. The Fed check uses the funds-rate move, not a human ranking. Joe Weisenthal's 4,005-speech post is a separate count from the 95-statement script.
| Check | Set | jsort | Other measure | Caveat |
|---|---|---|---|---|
| CommonLit easiness | 300 excerpts, -k 10 | Pearson r 0.824 · $0.046 | One Noul r 0.754 · SMOG 0.661 | Teacher reliability about 0.78, ceiling near 0.88. Run 2026-09-19. |
| FOMC hawkishness | 95 opening statements | Same-day rate move +0.46 | Next 180 days +0.37 | Top six are the 2022 50- and 75-point hikes. 4,005 is Joe's later post. |
Judging other output
Same job, different oracles and sample sizes. LangChain froze five weather traces and used a human label. Lemkin compared Jev with Sonnet on Connect pairs and posted a false-admission count. Every asked Jev 21 writing questions and did not put an LLM judge on the same 777 cells. JevBench's hard tier is a different rubric, written by Opus 5 and GPT-5.6 Sol. RZ recorded 100 synthetic security spans five times each.
| Who | Set | Jev | Against | Caveat |
|---|---|---|---|---|
| LangChain | 5 traces × 100, does_pass | 500/500 vs human | Terra 99.8%; Luna 96.4%; Sonnet 4.6 80.0% | One reviewer. Quality variance is the other claim. |
| Lemkin / SaaStr | 600 judgments | 70.5% vs third-model referee | Sonnet 77.5%; 32% vs 13.5% false admissions | Thread only. No prompts, no repo. |
| Every | 37 docs × 21 questions | 777 answers in <0.7 s | later 12-passage check vs Fable 5.1, 6 of 7 defects | No LLM judge on the 777-cell grid. |
| JevBench hard tier | 220 frozen items | 74.1% | Luna 94.5%; SemIf 59.5% | Composite still ranks Jev above Luna on cost. |
| RZ / security triage | 100 synthetic cases × 5 | 65.3% balanced accuracy · 0.57 s | Terra 75.1% · 10.23 s; Opus 91.2% · 13.83 s | Labels by one author. Safe-class recall 45.5%. CLI vs API latency. |
Legal-set loops
The host enumerates actions the runtime can take. Jev picks among those options. Illegal moves stay out because they were never listed. These are demos, not a shared computer-use eval.
| Demo | Set | Figure | Caveat |
|---|---|---|---|
| Browser Use | Zurich to London Google Flights | 7.1 s · $0.0039 | Repo notes; clip at 1x. We did not replay it. |
| Stagehand / jev-browser | Computer use from an a11y tree | $0.001, near-instant (Jeong) | Author post, not a Browserbase lab table. |
| Tetris | Same 200-piece sequence, legal landings | 9,200 pts · ~300 ms · 0 illegal | One run. Haiku 8,900 / 1.52 s; Gemini Flash-Lite 9,000 / 1.13 s. |
| JevPilot | Sampled steering paths in Three.js | up to 4 requests/s; $1.24 / 15,000 | A game. Geometry stays in the browser. |
| macOS Loop | Six native GUI tasks | 6/6; Finder 9 files in 7.39 s | Small functional sample. An earlier Calculator run failed. |
| Jevis | Flutter integration_test | no published pass rate | Typed strings come from the test author. |
Context filters
Tran argued for scoring tool output and dropping it. AstroHan kept the agent and the model fixed and reported whether the task still passed. One run per arm.
| Who | Set | With Jev | Without | Caveat |
|---|---|---|---|---|
| Tran / fast-jev-compaction | Keep-or-drop tool calls | plugin and library | summarization (the default it replaces) | No published pass-rate table in that repo. |
| AstroHan | FrontierHarness 30, deepseek-flash | 25/30 · meriyah 49/49 | 22/30 · meriyah 0/49 | Two of 30 the filter lost. 126m vs 123m input tokens. Thread only. |
Router tier on a labeled set
AgentOS compared an opt-in Jev strategy with its default on-device MiniLM classifier. Same 121 turns, gold labels R0 to R3, classifier only, two repeat runs identical. Blank cells are blank on the chart. A same-day follow-up says it sharpens the R2 and R3 wording. This is not a JevBench score.
| Metric | Local MiniLM | Jev | Caveat |
|---|---|---|---|
| Accuracy | 0.355 | 0.901 | 121 turns. Vietnamese, Chinese, English. |
| Under-routing | 0.413 | 0.066 | Gold is the project's own R0 to R3 labels. |
| Over-routing | 0.231 | 0.033 | Jev is opt-in. Turn text leaves the machine. |
| English / Vietnamese / Chinese | 0.513 / 0.312 / 0.250 | 0.946 / 0.938 / 0.806 | Chinese is the low Jev slice on this chart. |
| Cost and time | $0, on device | about $0.005, about 0.9 s per call | 121 cases. Macro-F1 0.894 and ECE 0.062 are Jev-only on the chart. |
Laya, two machines
Same weight families, different runtimes. The T4 times are from the Laya README cited in the earlier story. The M3 Max times are from the laya-mlx README. Neither row is a score against hosted Jev. laya-mlx also says the selected answer matched upstream on 63 of 63 validation questions.
| Checkpoint | Author T4 | laya-mlx M3 Max | Caveat |
|---|---|---|---|
| English, 421M | 39.5 ms, one question | 13.42 ms P50, 13.92 ms P95 | Different machines. MLX port, FP16, load excluded. |
| Multilingual, 322M | 32.8 ms, one question | 7.39 ms P50, 7.79 ms P95 | Batched multilingual on the T4 note was 7.2 ms per question at 10 questions. |
Picking a cube turn
maxli asked Jev for the best of 18 face turns, forty times, then removed the experiment. Kociemba in code is the solver that ships. A random walk is the other number in the same README note.
| Who picks | Distance | Time | Caveat |
|---|---|---|---|
| Kociemba, in code | at most 22 moves | about 20 ms | cubejs wrapper. This is the solve button. |
| Jev, best of 18 | 22 to 22 | forty calls | Experiment removed. The coach asks whether to speak instead. |
| Random walk | 22 to 21 | same note | One comparison the README prints beside the Jev run. |
Same Flights task
Zurich to London, one way, one adult, economy, stop when results are visible. The Jev column is jev-ultrafast's published measurement. dejevu did not rerun it. dejevu's own runs are September 22 on an M3 Pro, and the Sunday it searches is October 18 2026 because September 20 has passed.
| Agent | Verified | Time | Calls and tokens | Caveat |
|---|---|---|---|---|
| jev-ultrafast, Jev 1.13 + Mercury 2.5 | 3 of 3, published | 7.09 s | 17 calls, 84,650 in, about $0.0036 | Other machine. The earlier clip on this desk said about 7 s and $0.0039. |
| dejevu, llama-3.3-70b on Groq | 3 of 3 | 5.63 s | 10 calls, 15,071 in, $0.0092 | Faster here, and about 2.5 times Jev's list price on this route. |
| dejevu, same Llama on DeepInfra | 2 of 2 | 37.5 s | $0.0020 | Cheaper than the published Jev cell, and much slower the day they measured. |
Typed decisions
meraGPT's September 22 blog, one run, zero-shot for Decider 1 and for the Jev row. The fitted encoders were trained on each workflow's labels. Reference labels are the average of three teacher samples. The blog says 400 cases and 2,000 decisions. The Hugging Face viewer lists 1,600 rows across four subsets. We did not recompute the table.
| Model | Accuracy | KL | Brier | Caveat |
|---|---|---|---|---|
| Prior, ignores the input | 0.470 | 0.347 | 0.189 | Floor on the blog's table. |
| ModernBERT-base, fitted | 0.646 | 0.223 | 0.119 | Trained per question. Decider 1 did not get that. |
| Jev 1.13.0 | 0.727 | 1.442 | 0.148 | Choice gap vs Decider 1 is 0.013. The KL gap is the wide one. |
| Decider 1 (sd-1) | 0.768 | 0.096 | 0.052 | Hosted. 4,096 token cap. At most ten choice labels. $0.03 / 1M input. |
Banking-feed categories
Peter's September 22 post. 612 sanitized rows, described as a harder subset of about 33,000. This is not Banking77. Gemini costs are estimates and exclude thinking tokens. No category list was printed.
| Model | Accuracy | Time | Cost | Caveat |
|---|---|---|---|---|
| Jev 1.13 | 36.1% | 6 s | $0.023 | Closest to Gemini 2.5 on accuracy. Fastest of the three. |
| Gemini 2.5 Flash | 37.3% | 95 s | about $0.088 | Cost is an estimate from the text-token budget. |
| Gemini 3.7 Flash | 42.5% | 78 s | about $0.180 | Highest accuracy on this private slice. |
Hold'em clocks
Two published timings for Jinzhengxu/poker-table, plus the README's language-model baselines on the same 46 spots. The live pill and the spot bank are not the same measurement. The bb/100 table elsewhere in that README is a rule-policy ablation.
| Clock | Time | What was measured | Caveat |
|---|---|---|---|
| Live pill, September 22 post | 545 ms average, $0.002 a hand | The table the post links | We did not open poker.ccswitch.online. |
| Spot bank, Jev | about 1 s median | 46 spots, 3 repeats, OpenRouter | No accuracy percentage. Three passes under a cent. |
| Spot bank, single-shot LLM | 6 s | Same spots, reasoning model | README says Jev's range read matched it. |
| Spot bank, tool-loop agent | 35 s | Same spots | The path Jev replaced in front of the action clock. |