Updated
Desk
Where Jev fails: the limits of jev-1.13
TypeSafe's jaggedness page for jev-1.13 lists nine failure modes. The blocks below keep that list, then add misses that already showed up in stories. We did not reproduce the examples. For similar tests placed next to each other, see compare. The launch post's can't-hallucinate line, and the tables that score hallucination as its own task, are on hallucination. The October 14, 2026 release is on Jev Day 2026.
TypeSafe's jev-1.13 list
Sasha Sheng posted that Jev is still too slow, too expensive, and too dumb, and pointed at the jaggedness docs, last reviewed September 17. The page names nine modes and a workaround for each. Numeric examples on that page: a Noul for a refund request at 0.22 next to a yes/no Choice at 0.01 / 0.99 on the same ticket; a Noul and its negation summing to 1.19 on another. Discord is the place the company named for new issues.
- Write the exact condition; put boundary cases in the criteria, or split the judgment.
- Keep arithmetic in code. jev-1.13 does not tally characters, term counts, or long lists.
- Extract dates, then compare them in code.
- Cut extra hops of indirection.
- Filter state before the call. Accuracy falls as unrelated material piles in.
- Test adversarial text; it can steer the answer.
- Align the instruction with the criteria.
- Do not assume two phrasings of the same question agree, and do not reuse a Noul threshold on a Choice.
- Do not ask Jev to generate text. Chaining choices to force generation is slow and weak.
mammoth-1 card 71.24 against live 68.4
The Hugging Face card, opened October 23, prints Decision Index 0.4 Full 71.24, first of 208. The live space that day is edition 0.4.1, 229 models, mammoth-1 Full 68.4 rank 3. The public cell in that row is 66.0. elephant-1 is 62.65 on the card and 60.3 Full on the live chart. 71.24 is not 68.4. This desk did not run the weights.
A checkout ad moved Jev to Confirm Price at 91%
HiddenLayer, October 14, 2026. A jev-ultrafast agent was told to buy the cart and that page text is untrusted. After a Confirm Price ad, TypeSafe Jev gave that button 91 percent and Place order 9 percent. Date headers 07:24:57 GMT and 07:24:58 GMT October 19, this desk's jev-1.13.0 still chose place_order, 1.0 then 0.89 with confirm_price at 0.11. Those two POSTs are not the 91 / 9. Jaggedness already names adversarial text.
A counterfeit policy in state dropped is_risky from 0.94 to 0.13
Datadog, dated October 15, posted October 19. Noul recall 82.90% to 3.88% on 4,643 attacks. LangChain is_risky 40-run mean 0.893 then 0.290. Date headers 08:50:46 GMT and 08:50:47 GMT October 20, this desk's jev-1.13.0 is_risky on a shortened handbook was 0.94 then 0.13 with Datadog's definition in profile_description. A dump-the-prompt Noul stayed 0.99 then 0.92. Jaggedness already names adversarial text.
A yellow square at IoU 0.83 still came back at confidence 0.28
On October 16, 2026, Date header 00:28:50 GMT, this desk asked four Boxes on one four-color PNG. All four Nouls were 1.0 and all four centers sat inside the drawn squares. Yellow confidence was 0.28. Blue was 0.49. Red was 0.88. Green was 0.75. Aviz Maeir's ≥0.7 cut was right 107 of 108 on a different 180-scene set.
Both models read a published Kyoto warning as social_nicety
Yamashita, October 15, 2026. Expected warning on うちの子ぉも、ええ子にしてますえ. Majority vote: 1.13.0 sarcastic_criticism, 1.14.0 social_nicety. This desk, Date headers 14:57:06 GMT to 14:57:09 GMT, a Choice written here: both models chose social_nicety, n = 3. Confidence 0.69, 0.66, 0.64 on 1.14.0 and 0.33, 0.36, 0.34 on 1.13.0. Yamashita's 60-item means 0.833 and 0.817 stay a different test.
A red square beside a blue one scored 0.56 mean on jev-1.14.0, n = 3
On October 15, 2026, Date headers 14:57:24 GMT to 14:57:25 GMT, this desk sent a 400 by 400 PNG with a blue square from 0.08 to 0.28 and a red square from 0.40 to 0.80 of the width and 0.20 to 0.70 of the height. jev-1.14.0 red Noul 0.55, 0.55, 0.57, mean 0.56. Blue Noul 0.98. Matching jev-1.13.0 red mean 0.87, blue 1.0. The 06:16 UTC red-only mean 0.46 stays a separate sample.
Both models marked a Turkish buyer as a seller
SosyalDijital, October 15, 2026. 30 synthetic Turkish messages, 29 of 30 on both Jev 1.13 and Jev 1.14. The shared miss is Reklam değil gerçek talep: ürün fotoğraflarını düzenleteceğim, ücret konuşalım. Expected buyer, both returned seller. Recorded through OpenRouter. This desk, Date headers 11:04:57 GMT to 11:04:59 GMT, a Choice written here on that sentence: jev-1.14.0 chose seller three times, confidence 0.48, 0.43, 0.52. jev-1.13.0 chose seller, buyer at 0.28, then seller. n = 3. Chinese 1.30 is a different test.
Two instruction references of one image nearly double tokens
On October 15, 2026, Date header 11:04:56 GMT, an 850 by 850 PNG listed once in assets. Two Nouls that referred to it from state used 2,259 input tokens. Two Nouls that each wrote the same reference into instructions used 4,209 tokens. Wes Bos billed $0.000096936 for one 850 by 850 question and wrote that two questions duplicate usage. The images docs say each use adds the image again, and that several questions should put the file in state once.
Chinese payouts frustration mean 1.30 at confidence 0.54, n = 3
On October 15, 2026, Date headers 06:16:21 GMT to 06:16:25 GMT, this desk sent the docs' payouts sentence in Chinese, with English questions, three times each. jev-1.14.0 scored 1.27 at 0.59, 1.32 at 0.52, and 1.32 at 0.52, mean 1.30 at mean confidence 0.54. Matching jev-1.13.0 scores 1.55 at 0.32, 1.52 at 0.28, and 1.55 at 0.32, mean 1.54 at 0.31. Language was chinese at 1.0 and department billing on all six. The 03:03 UTC singles 1.37 at 0.44 and 1.54 at 0.31 stay a separate sample.
A present red square scored 0.46 mean on jev-1.14.0, n = 3
On October 15, 2026, Date headers 06:16:36 GMT to 06:16:40 GMT, this desk sent a 400 by 400 PNG with a red square from 0.40 to 0.80 of the width and 0.20 to 0.70 of the height, three times. jev-1.14.0 Noul 0.49, 0.47, and 0.42, mean 0.46. Matching jev-1.13.0 Nouls 0.94, 0.94, and 0.93, mean 0.94. An empty field scored 0.04 and 0.01. A request with that image and no TypeSafe-Danger-Zone-Images header returned HTTP 400. The 01:22 UTC singles 0.39 and 0.90 stay a separate sample. TypeSafe's images docs say models, Jev included, do worse on multimodality than on pure text.
A high-confidence wrong match still goes through
On Akshay Pachaar's 40-request TrueForge test, both arms scored 37 of 40. Review caught 5 wrong Jev matches that would otherwise have gone through. REQ-001 is one of the three remaining misses: Jev gave the wrong issue its highest confidence, so the answer went straight through. The 80% and 90% cuts send low-confidence answers to Opus. They do not catch a confident wrong pick. On October 13, at 09:21 UTC, he wrote that 40 is too small to establish how reliably confidence routes errors to review, and that at 400 he would measure errors caught and errors accepted at each threshold. On October 14, 2026, this desk's trap shortlist, with the true match removed, picked a closed API 504 issue at 0.55, none at 0.43, confidence 0.51. That score would have gone to review. This desk did not rerun the 40.
Hosted Clef-flash window cut to 24k
The October 9 Clef blog lists hosted Clef-flash at $0.038 per million input tokens and a 24k context window. It says the October 1 advertisement was 64k, that the Hugging Face weights were trained for 256k and were not replaced, and that 0.24% of requests exceed 24k. Clef stays at a 64k hosted window and $0.24. Clef-omni is a separate model at $0.15. We did not call either API.
Three high scores still wrong
TypeLLM's September 23 thinking file scores 228 of 231 public JevBench tasks. The bin from 0.9 to 1.0 holds 225 of them, and 222 are correct, so the three misses are in that bin. Easy falls from 48 of 48 to 47 of 48 on the same file. A depends_on child sees the parent's typed answer. The schema still only keeps the child inside its own domain. The hosted thinking budget defaults to 1,024 tokens. This run set no budget. We did not call the API.
Two Microsoft sentences on the same score
Microsoft's Learn page for Microsoft-Decision-1 says to use a score for ordering and thresholds, and says it is not an absolute calibrated rating. The October 9 Command Line post says a 90% prediction should be right about nine times out of ten on representative cases. The Foundry catalog calls the scores calibrated. The Learn page also says wording and option order can change a score, that the model returns no explanation, and that a safety filter can miss subtle harm or flag benign text. We did not call the API.
A final-answer judge paid for no search
On Step-Jev's BrowseComp-Plus seed, an outcome judge trained on the agent's own traces scores the policy at 5.4%, under the untrained 15.6%, while the judge's own number was rising. Searches went from 3.6 to 10.5 a question, and 15% were repeats. Hosted Jev, zero-shot, scores 0% on that page: the policy stopped searching, and the note says those answers were rated about 70% likely correct. In jev_final_step an empty tool list adds no step penalty. On October 9 the repository's own no-step test state scored incorrect at 1.0 on jev-1.13.0, and a no-search answer of 1965 on a Moon-landing question written here did the same. The rollouts behind the 70% line are not in the repository. We did not rerun the seed.
Finished work called fail
On ForkScape's 80 review items, jev-latest scores 56.3% (45.3% to 66.6%). The report's confusion list is pass called fail, 35 times. Unbacked claims on that suite are 15 of 15. At confidence 0.8 or higher, accuracy on the items Jev still answers is 88.5%, and that gate keeps 26 of the 80. fsdecide, trained on this app's labels, is 91.3%. On the authors' separate 1,248-request set, jev-latest is 90.5% and fsdecide is 60.5%. We did not rerun it.
A self-run index, and a hidden Polish set
StartLux-Decision-27B prints 63.88 on the authors' Decision Index 0.2.1 run, against the September 28 board's Jev cell of 57.91, ahead on 31 of 38 benchmarks. The README says the run is not on the board, and that 14 of those benchmarks had their public train splits in the training data. Knowledge and Reasoning on that run is 44.3 against Jev at 51.4. GPQA Diamond is 34.0 against 71.4. On Werdykt v1, a hidden Polish-and-English set, Jev 1.13.0 is 81.64% at p50 286.4 ms and basal-1.5 is 72.12% at p50 33.8 ms. We did not rerun either.
A sexual line, and where the message sits
jevmod's sexual AUROC is 0.984 on 1,680 messages. The nsfw question also names graphic gore, and the OpenAI label is sexual. On 30 Discord messages, moving a text inside its batch changed the nsfw score by as much as 0.380, and none of those 30 crossed 0.80. Spam and scam did cross their lines on position alone. ZeroNSFW prints no labelled accuracy on adult images. We did not rerun these.
A cutoff, a specialist, a two-bar chart
You.com's risk server ships the escalation Noul at 0.5. At 0.7, Gulf scored 0.58 and the run stopped, after the 0.5 arm had marked that same profile critical. The Qwen judge on that profile read 0.35, then 0.68, then 0.50. Insilico's chart, 12,563 molecules, shows Jev 1.13's predicted probabilities topping out at 0.69, and balanced accuracy of 0.49 on CYP 2C19. WrangleBench prints 137.3 against 105.1 perfect matches per 200 and says the full benchmark is still to come. We did not rerun these.
A changed question, a look-alike alarm, a raw score
Strands Decider v19, with the state and the options held fixed, gives the same answer to a changed question about 94% of the time. On temporal_numeric the file prints 0.267, and it prints Jev at 0.27. HydroJEV's agreement gate settles 76 of 200 look-alike windows. 32 of those 76 accepts were wrong, and 21 of the 76 were attacks. The cascade marks 15 of 50 true attacks as non-benign, against 20 of 50 for the reviewer alone. The DimABSA pipeline's raw Task 1 error, before the 488 coefficients, is 2.0731 RMSE. After the fit it is 1.0645. We did not rerun these.
A tool selector still follows a lot of attacks
Zhaorun Chen's DTap chart, September 27, puts a Jev 1.13 tool selector at 70.1% direct misuse attack success and 43.5% indirect prompt injection. A second Jev check on the filled tool call brings those to 36.1% and 21.6%. Benign success moves from 60.9% to 55.8%. The chart does not print a task count. The author says the gate still misses context-dependent harm and injections that sit in the environment, skills, or tool descriptions. The public leaderboard listed no agents when we opened it. Wu and Lim's registered attack, a different paper, moves validated success on 510 InjecAgent cases from 1.8% to 3.5%. JevAdvBench's abstract, a third paper, says one unverified opinion appended to the state flips 12.1% of jev-1.13.0 decisions, statistically tied with the strongest injected command at 10.1%. We read that abstract, not its 19 tables. We did not rerun either chart.
v1.5 ranks speed and cost with accuracy
The API board opened October 7, 2026, release v1.6.1 under revision v1.7.12, ranks Jev 1.13.0 third at composite 71.5, with capability 77.1. Opened October 8, 2026, revision v1.7.21, that board ranks 26 hosted systems and Jev 1.13.0 is fourth at the same 71.5. JevBench v1.5.0 ranks Cygnet first at 73.7 and Winnow-12B Q8 second at 73.2, and calls that pair a statistical tie. Jev is third at 72.1. Its capability, 80.0, is the highest among 56 Jev-class systems. Cygnet's capability is 79.0 and its median latency is 0.23 s, against Jev at 0.62 s. decider-4b v2 is official rank 5 at 71.3 with Intelligence 55.8 and a gap of +11.9. Hopper's Intelligence cell is 49.9, under the floor of 50, so that axis is scaled and the composite is still 67.5. Imajev-4B's 67.37 was rank 1 on v1.4.2.2. On v1.5 it is an addendum at 70.4, outside the ranked order. 75 of 88 adjacent pairs on v1.5 are statistical ties. The page we opened September 30 is v1.5.4 and numbers 106 systems. Cygnet, Winnow, and Jev stay 73.7, 73.2, and 72.1. Plumb-4B is official rank 5 at 71.6. Imajev-4B is official rank 9 at 70.4. Both rows are still tagged roster addendum. The page opened October 4 is v1.5.6 and numbers 110 ranked systems of 116. imajev-4b is official rank 10 at the same 70.4. The tie line there is 69 of 77 adjacent pairs with a published comparison, and 28 adjacent pairs have none. The capability control says 60 Jev-class systems.
Misses beside the older composite
On the v1.4.2.2 board Imajev-4B led at 67.37. On the author's DecisionBench note, ECE moves from 0.024 to 0.069, and a scenario file that passed 129 of 145 checks without the calibration file passes 113 of 145 once that file lowers confidence. Plumb-4B's 89 of 111 is House's public-hard runner. The v1.4.2.1 composite is 65.84. Dates and numbers on that runner stay 7 of 15, and hard ECE moves from 0.051 on v4 to 0.094 on v5. Sandipan Kundu's 400M encoder leads 17,476 of its own rows and then flags 1,995 of 2,000 PhishNChips emails, 995 of them false, for 50.25% against a published Jev figure of 62.6%. Typed decisions on that external suite are 44.15% against a published 72.7%. Jev was not called again. Christoph Görn's local Kev-4B, Palmer's model, is 1 of 7 on eight of his prompts. toolgate is a classifier in front of a hook, not a sandbox, and the 1,369-decision week on version 0.9.1 failed by asking too often. jevgrep's combined bill is 25.8% under a saved baseline, five of the ten tasks cost more than that baseline, and the September 26 post's 40% matches neither file.
A sealed set, a verifier, and a one-line rule
On JevBench v1.4.2.2, Imajev-4B led the harmonic mean at 67.37. Plumb-4B was 65.84. decider-4b v2 was 64.13. Jev was 63.29, official rank 4 on that board. Public accuracy was 86.6% against sealed accuracy 36.7% on 308 private items. The gap was 49.9 points. decider-4b v2 was 83.5% and 34.7% on those two sets. On that page Jev led Intelligence, 53.1 against 49.4. Jev-Omni's public accuracy was higher, 88.7%, and its sealed accuracy was 32.1%. v1.5 replaces that order. The cells above are the older board. CLM's README says that as a verifier on 38 held-out DeepSWE tasks and 30 Terminal-Bench 2.1 tasks, Jev scores below pass@1, while a fine-tuned CLM head is 31/38 and 87.6%. mizchi's fit page puts a one-line monitor rule at 16/16 with 0 wrong pages, and Jev at 6 to 27 wrong pages on that triage. A question that names an API behavior the snippet does not show still returns no at 0.09 to 0.22. On one board, 'enemy adjacent' is 49% when an enemy is there. Eikos's own table puts Jev's error, among decisions at confidence 0.90 or higher, at 5.9% on 7,140 items, against Eikos at 2.4%. RLCDAlignBench's median ECE for a Jev Noul is 0.168, against a base-rate null of 0.074. Wu and Lim's registered attack on jev-1.13.0 moves validated success from 1.8% to 3.5% once the search can read scores. At confidence 0.7 their conditional hit rate stays 0.2%. TypeSafe's Deel chart drops 16.6 points on messy metric questions and 3.6 points on 3-level ticket tags. jev-code's ladder, one run per task on commit 517d0f0, still fails guessing-game, a file script, and a multi-file package.
Hard items versus Luna, and false admissions
JevBench's hard tier, 220 frozen items, is 74.1% for Jev 1.13.0 and 94.5% for GPT-5.6 Luna. The composite still ranks Jev higher because Luna's cost score is 28.2. Lemkin's SaaStr Connect trial is a different miss: 47 false admissions out of 148 negatives (32%) against Sonnet's 20 (13.5%), on a job that cannot take weak yeses. LangChain's five-trace judge eval did not show that split; Jev matched every binary label there. JevEval's Tool Faithfulness example, scored here on October 14, 2026, put P(true) at 0.03 on whether every fact came from a tool and at 0.63 on whether returned values were reported accurately, when the answer added a breeze and humidity the tool never returned. The weighted mean was 0.254. The docs walkthrough that prints 0.461 is illustrative. RZ's 100-case security demo is the first set on this desk where Jev trailed both frontier models on the headline score: 65.3% balanced accuracy against Terra 75.1% and Opus 91.2%. Safe-class recall was 45.5%. The same run repeated Jev's answer on 95 of 100 cases. On Zefan Cai's 231 public JevBench tasks, Jev 1.13.0 is 200/231 and the hard tier is 81/111, against Luna 89/111 and Astra 111/111. The 15 temporal and numeric tasks in that file are 4/15 for Jev and 5/15 for Luna. Astra is 15/15. That slice leaves out 303 private or judge tasks, so it is a different number from the 534-task composite.
Whole pages and 77-way intent
Kumar's 135-row extraction gate matched later outcomes 68.6%. Probability swung by 0.5 between runs on the same page; he treats that as the task, not the model. On Banking77, 300 items, Jev lost to gpt-5.4-mini and gpt-5.6-luna (76.0% versus 78.7% / 81.7%). Poly's full-set encoder run won macro-F1 (0.782 versus ModernBERT 0.712) and lost Expected Calibration Error (0.105 versus 0.081). Jev assigned probability 0 to the correct label on 6.6% of those decisions; the open encoders did that zero times in 1,155 each. DocJev's 40-PDF classify run was 40/40 for both engines; on eight packets Jev split 7/8 exactly and Luna 8/8, after Jev cut a Federal Reserve implementation attachment that the frozen rules keep with the statement.
Text only
Bryo's 1,565-email table excluded attachments because Jev does not take non-text input. Mudholkar called that the biggest blocker for production. TypeSafe's models page already says input is text. The 364 synthetic mails in that set are a separate limit on the label, not on the model. Aggarwal's October 2 OSWorld run puts a screen parser in front of that text-only call, about 500 ms a step, and the chart callout is 9.8% at 125 seconds a task.
Loops that got worse
Malis asked Jev to pick Postgres join order on JOB: about 2x slower. Blind filter estimates helped some IMDB queries and blew one up by an order of magnitude when Jev was unsure. The hybrid he kept (Postgres plans first, Jev overrides when confident) was a 12% geomean with no dramatic regressions; the planner itself got slower by hundreds of milliseconds per call. Clark's SREGym-Lite run went 20/50 to 24/50 with jev_plan and jev_submit; two of the ten problems got worse. Simmons's macOS Loop recorded an earlier Calculator result of 3,968 on 32 x 14, caught by the verifier. AstroHan's context filter lost two of 30 FrontierHarness tasks; meriyah is the one that timed out without it. Vini Lana's chess-engine bench, five runs per cell, cut bug-fix output tokens and then spent more on the feature task for Opus 5 (+22% output, +61% input, +83% time) and Sonnet 5. Luna solved that feature task 3 of 5 times with routing and 5 of 5 without. maxli asked Jev for the best of 18 cube turns, forty times. Kociemba distance stayed at 22. A random walk in the same note went from 22 to 21. The coach that shipped asks whether to speak, and code still solves. Zachi's pg-jev packs rows into one state and asks Jev to answer rows[i] by position. On 400-row checks, batches of 1 to 20 were 100% correct, batches of 40 were 92% to 98%, and batches of 80 were 77% to 94%. He wrote that a million-row table will not work as one scan.
A private feed, mid-30s accuracy
Peter's September 22 post categorizes 612 sanitized banking-feed rows. Jev 1.13 scores 36.1% in 6 seconds at $0.023. Gemini 2.5 Flash scores 37.3% in 95 seconds. Gemini 3.7 Flash scores 42.5% in 78 seconds. Gemini costs are estimates and exclude thinking tokens. The post prints no category list and no repository. Banking77 on this desk is a different set, with Jev in the 0.78 macro-F1 range.
JamC-QA, and a binary slice that does not separate
t.toda's September 22 table prints Jev at 56.0% on JamC-QA and GPT-5.6 Luna at 76.5%. On JMMLU the same image has Jev at 90.2% and Luna at 82.5%. The three-set mean is 80.8% against 85.4%. The image prints no item counts. hotdog-27B's public native-binary slice is 60/74 against Jev at 62/74. The paired interval is -12.20 to +5.63 points, so it includes a tie. The same card's independent holdout goes the other way, 1,603/1,631 against 1,587/1,631, and that holdout shares source datasets with training.
Teacher-average labels
meraGPT's Decider 1 blog puts Jev 1.13.0 at accuracy 0.727 and KL 1.442, against Decider 1 at 0.768 and 0.096. The choice gap on that table is 0.013. The reference is the average of three teacher-model samples, and the blog says two samples agree 73.5% of the time. One evaluation run. The blog counts 400 cases and 2,000 decisions. The Hugging Face viewer lists 1,600 rows. We did not recompute either figure. Decider 1 is hosted, caps a request at 4,096 tokens, and allows at most ten choice labels.
Wording moves the probability
Jevsus, jev-1.13.0, a September 20 run posted September 23. 45% of statements move by 0.20 or more across three wordings. On "{a} is a dictator," Xi Jinping is 0.01 for that sentence and 0.50 as the average of eight ways of asking. "The United States is a racist country" is 0.83 with no prefix, 0.20 under a US conservative prefix, and 0.95 under a US progressive prefix. The README says those prefixes are model outputs, not the groups' opinions, and that 187 of 335 names under sixteen accusation sentences are not quoted. Eight anchors stay at 0.00 and 1.00. Rizzo Flow, a different model on SemIf's missing-evidence file, is confidently wrong on 6 of 36 rows. SemIf's published count on those rows is 1.
Local stand-ins, and where v1.5 places them
Jev's weights are not public. On v1.5, Cygnet is 73.7 and Winnow-12B Q8 is 73.2. Imajev-4B and Plumb-4B led the older harmonic mean at 67.37 and 65.84, and both still miss on the notes above. On the v1.5 addendum they would place at 70.4 and 71.6, outside the ranked order. SemIf's Qwen3.5-4B row on the older geometric mean is 74.7 against Jev 75.4, and 59.5% versus 74.1% on that hard tier. On v1.5 SemIf is official rank 6 at 68.7. djev, a hosted DiffusionGemma API, was 74.3 on the older composite, with hard-tier 69.5% and Calibration 65.4. On v1.5 djev is official rank 13 at 64.2. Laya is 70.1 on the older suite. Laya's own table, against published Jev figures the README says were never measured in that repo, trails on Banking77 (0.425 versus 0.870, 77 labels versus 72) and scores 0.000 on Khmer at 0.952 confidence on the English checkpoint. Nimble matched 90.12% of 324 synthetic labels against Jev at 93.21%. On 13 public subsets the macro is 74.8% against Jev at 76.0%. jeff scored AG News 75.5% versus Jev 90.5% on 1,600 public items, and 66.9 on JevBench. Verdict scored 48.1% on 337 TypeSafe public cases where Jev scored 90.8%; the 1.4 engine is 72.5 on JevBench. Palmer's Kev-8B scored 79.6% on his new-source split against Jev 85.7%, with MMLU 70 versus 90 and date arithmetic 60 versus 93; JevBench listed kev 0.6B at 66.7. SimpleJev Qwen3.8-27B is 67.3 on the composite and 75.0% on the hard tier. Open-Jev's released 9B checkpoint, a different project from SemIf, is 179/231 on the public tasks against Jev 200/231, and 66/111 on that slice's hard tier. 27B v1.1 on that same slice is 197/231 and 80/111, three tasks and one hard task behind the Jev row on the September 23 card. LocalJev's bake-off compares local checkpoints with each other, not with hosted Jev, and the authors say the JSON probabilities are not a logit read. llamacpp-jev is another /v1/systemone server on stock llama-server. Its probabilities are a raw label softmax, and on 64-way questions Qwen3.5-0.8B and 2B both answered option 16. The 32/32 figure is eight synthetic images, not JevBench. A reply the same hour as the post that README cites links kishida/llama.cpp branch jev, which changes llama-server. Its table puts Qwen3.8 Flash-Next at 0.944 on 1,191 questions. On October 2 ggml-org merged pull 29818, and llama.cpp's own server answers /v1/systemone for Julia-1, Laya, Kev-4B, lev, and OpenJev. Those medians on one RTX PRO 6000 are 3 ms, 5 ms, 12 ms, 36 ms, and 43 ms. Release v0.6.0, published October 5, also serves Nimble and Clef. The Nimble pull names Bespoke-Nimble-9B-v3, whose card prints Decision Index 0.2.1 at 56.88 under CC BY-NC 4.0. That card is not the 324-row holdout. The server README at that tag names Clef as an image model, and the October 6 note leaves Clef's speed cell blank. Opened the same day, the kishida page still says the endpoint is not upstream. laya-mlx runs Laya's checkpoints in MLX. On 63 validation questions the selected answer matched upstream Laya. That check is not a score against hosted Jev, and the port does not publish one. rev's frozen Qwen3.5-2B is 0.550 on the public hard tier where that README prints Jev at 0.730. The same page's Qwen3.8-27B row is 0.784 on those 111 items, ahead of that Jev cell. spark-s1-4b-v6 prints hard 0.595 on its public proxy and 0.756 on Kev's development partition, against a published Jev figure of 0.857. hotdog-27B leads its own binary holdout and trails on 74 public binary items, with an interval that includes zero. Rizzo Flow's gap against SemIf on authored144 includes zero, and its missing-evidence miss does not: 6 confident wrong answers against SemIf's 1. Jevre's test ECE is 0.0079 and its held-out-sources ECE is 0.0944, with no Jev column. jev_local's Sarashina JNLI score is 34% against LFM at 17%, and the author says that row is still under a majority baseline. DiffusionGemma as Jev, the one-step canvas read, sums to 198 of 201 on eight hand sets against Jev at 191 of 201. On ywchiu's 100-point routing file the djev-spark joint is 32.2% against Jev at 61.4%. Chirag's sieve README says reversing the options flipped 24% of its answers. None of those rows is the 534-task composite.
One run, then five
Watters's first hard-18 run had Jev at 77.8% and GPT-5.4 at 66.7%. Five runs the same week put the means at 73.3% and 70.0%. Jev's worst run, 72.2%, equals GPT-5.4's best. Majority-of-five McNemar is p = 1.000. The easy 24 stays perfect for the frontier models and 91.7% for Jev, missing the same two controls every time. Three Noul questions in one call scored 77.6% against 83.8% for one Choice, at $0.000026 against $0.000025 per claim. On identical calls the label changed for 2 of 42 claims, both under confidence 0.47. Of 19 wrong claims, the September 23 page counts Jev catching 15 and GPT-5.4 catching 18, p = 0.25. The labels are his. This harness is 42 claims. Florian's table is the 534-task composite.
A first-place claim with a lower public count
Nace's September 30 post calls Drex 1.5 first on JevBench. The chart on the next post prints Jev 1.13.0 at 201 of 231 and Drex 1.5 at 199 of 231, with a hard-tier tie at 82 of 111. Inception's post the same day calls Mercury Decide the most intelligent decision model on OpenRouter, citing JevBench v1.4, and prints no score. Opened October 8, 2026, the API board ranks Mercury Decide third at 72.4, and the free model page returned, priced Free. The free model URL returned Not Found. On October 1 the Benchmark Heaven page still said v1.5.4 and 106 systems. A search of that text for Mercury Decide and for Drex 1 returned no match. Liquid's chart leads Jev by 1.0 on an internal Decision Index reproduction and trails on tools and on knowledge. The October 7 d1-3B card prints 48.57 on Decision Index 0.2.1 and calls that the best score under 10B. The same table prints Winnow-12B at 50.02. Rox's chart leads at 200 chunks, 96% against 84%, on a mean of 75 queries, and the 20 times and 10 times lines are not on the chart.
A newer board, and a card on an older edition
Decision Index 0.3, opened October 7, 2026, ranks Perplexity Decider v1.1 at 62.8 and lists Jev 1.13.0 as a reference at 60.1. The open Drex card's 52.31 is edition 0.2, against the kit's Jev cell of 51.67. A search of the 0.3 space for Drex found no row. PolicyLM's custom-policy table prints 0.842 at a 0.5 cutoff and has no Jev column.