JevBench v1.5 ranks Cygnet 73.7 and Jev 1.13.0 72.1

Florian S posted JevBench on September 19 as a benchmark for models that take state plus a fixed label set and return a typed answer with probabilities. The interactive table is on Benchmark Heaven. The harness, tasks, and scoring live in fstandhartinger/jevbench. The README says the project is not affiliated with TypeSafe.

v1.2.2, the revision on the page we fetched, runs 534 decisions per system. Easy 72, standard 96, judge 146, hard 220. The published score is a geometric mean of four axes at 25% each: Intelligence (weighted accuracy), Calibration (hard-tier ECE and fidelity to gold distributions), Speed, and Cost. One weak axis pulls the number down.

The ranking Florian quoted in the first posts was Jev 1.13.0 at 75.3 and SemIf, a Qwen3.5-4B open rebuild, at 74.6. The current table still has those two rows. It also puts classifier.dev’s fast tier first at 84.8. The README says that tier is Jev behind another API: Intelligence 90.1 versus Jev’s 90.4, p50 0.39 s versus 0.65 s from the bench’s server, and a Pro plan the authors price at full use as about $0.0033 per 1,000 decisions against Jev’s measured $0.041. That is a hosting and billing row, not a second model.

GPT-5.6 Luna (low reasoning effort) has the highest Intelligence, 96.8, and the highest hard-tier accuracy among finished runs, 94.5% against Jev’s 74.1%. Its composite is 66.0, rank 12, because Cost is 28.2 at $0.247 per 1,000 decisions. DeepSeek V4.1 Flash is similar: Intelligence 96.1, Calibration 96.7, Cost 17.1.

Hard items were written by Claude Opus 5 and GPT-5.6 Sol, then cross-reviewed and hashed before any system ran. 111 of 220 are public. Self-hosted and demo endpoints have their latency multiplied by 2 and 0.15 s added; the README calls that an assumption, not a measurement. Production APIs, including Jev, are left raw.

A Benchmark Heaven reply on the same thread says TypeSafe called Master Customer Agreement section 2.3(f) an outdated pre-launch constraint they are fixing. We did not find a TypeSafe post that says that.

Open rebuilds fill most of the middle. SemIf 74.6, djev 74.3, Laya 70.1, jeff 66.9, Bespoke Nimble 9B 63.5. Several rows are estimated cost, not a bill. We did not rerun the suite.

On September 24 Benchmark Heaven posted v1.4.1. The page we fetched ranks 77 systems, with 5 more shown and not ranked, on the same 534 public decisions plus 308 sealed decisions. The sealed text stays private. The score is now an equal-weight harmonic mean of Intelligence, Calibration, Speed, and Cost. Intelligence is 80% of the v1.3 axis plus 20% chance-corrected sealed accuracy. A public-minus-sealed gap wider than 25 points reduces Intelligence. Speed and Cost keep the v1.3 measurements. The frozen top five on that page are Jev 1.13.0 at 63.3, JevK5 v0.2.0 at 62.0, Hopper at 59.4, Winnow-12B Q8 at 55.6, and reflex 4B at 54.0. The GitHub README prints those as 63.29, 62.04, 59.43, 55.58, and 53.99.

The axis table on the same page puts Jev at Intelligence 53.1, Calibration 76.3, Speed 83.3, Cost 52.0, public accuracy 86.6%, sealed accuracy 36.7%, a gap of 49.9 points, $0.040 per 1,000 decisions, and p50 0.65 s. JevK5 is 48.9, 74.5, 91.1, and 59.5, with public accuracy 85.3%, sealed 33.1%, a gap of 52.2 points, and an estimated $0.022. Its latency cell is blank. The JevK5 post says Jev leads Intelligence 53.1 to 48.9 and Calibration 76.3 to 74.5, and JevK5 leads Speed 91.1 to 83.3 and Cost 59.5 to 52.0. The page’s note on JevK5 says the author reports no JevBench items in training, and that a scan found one generic instruction shared by 8 public hard items.

Jev-Omni is rank 7 at 51.3. The axes are 46.8, 64.1, 81.5, and 53.0. Public accuracy is 88.7%, sealed is 32.1%, the gap is 56.6 points, estimated cost is about $0.037 per 1,000, and p50 is 0.22 s on a RunPod GPU. The post says the hard tier is 75.0% against Jev’s 74.1%, and the scores are 51.34 against 63.29. The page’s note says the weights are akhilaaa3/Jev-Omni, a merged Gemma-4-12B with a 256-way head, run offline so the sealed text did not go to an operator endpoint. The compare widget on the URL in that post rendered Jev against JevK5 when we opened it, so the 75.0% figure is the post’s, not a second copy we read off the page.

GPT-6 Luna appears twice, at 35.1 and 33.3, with Intelligence 95.8 and 97.4, public accuracy 99.1% and 99.6%, and sealed accuracy 92.9% and 95.5%. Those rows are instruction models on the new formula. classifier.dev is an honorable mention at 70.8 and is not ranked. SemIf is rank 9 at 47.7. Laya is 30.3. A 63.3 here is not a drop from the 75.3 geometric mean above. The formula changed, and a fifth of Intelligence now comes from the sealed set. We did not rerun v1.4.1.

On September 24 at 23:38 UTC Florian posted v1.4.2. The order in the post is decider-4b v2, Jev 1.13.0, JevK5, Cygnet, Hopper. The changelog dated September 25 says the release adds eleven systems, finishes swanOne’s sealed run, and leaves the scoring formula and every v1.4.1 measurement unchanged. The README prints 64.13, 63.29, 62.04, 61.76, and 59.43, on 93 systems of which 89 are ranked. The live page rounds the top of that list to 64.1, 63.3, 62.0, 61.8, and 59.4.

The decider-4b v2 page shows Intelligence 49.4, Calibration 75.0, Speed 92.9, and Cost 60.9. The aggregate table puts public accuracy at 83.5% and sealed accuracy at 34.7%, a gap of 48.8 points, with p50 0.02 s. Jev on the same table is still 86.6% and 36.7%, gap 49.9 points, p50 0.65 s, $0.040 per 1,000 decisions. The row note says the weights are Mapika/decider-4b revision 7ab294cbdf6be6ac17fc818c10cdead744393d92, temperature 1.935, served by decider-ai 1.2.2. The run was offline on an evaluator-owned Lium RTX PRO 6000, through the author’s server on loopback. The main table’s endpoint column says RunPod GPU. Cost is labelled estimated: DeepInfra’s Qwen3.5-4B list price of $0.03 per million input tokens, read September 24, times the server’s own input tokens. It is not a GPU bill. The note also says the author disclosed that 8,000 of the v2 LoRA rows come from generators written from the published names of the ten sealed families, and that the private stage-2 rows were not audited for overlap with public items.

The follow-up post says Jev leads intelligence, 53.1 against 49.4, and calibration, and that decider-4b v2 wins on speed and cost. It says about 5 times faster and about half the price. Half the price matches the $0.020 estimate against $0.040. The latency cells, 0.02 s and 0.65 s, are a larger ratio than 5. We did not find a second clock in the post that produces 5. The next morning Florian wrote that Jev is first on raw intelligence on both this board and the Decision Index, and that decider-4b v2 leads the JevBench composite because of speed and cost. We did not rerun v1.4.2.

On September 27 the changelog moved the composite twice. v1.4.2.1 adds Plumb-4B at 65.84 and reprices that row’s estimated input at EmpirioLabs’ bookable Qwen3.5-4B rate, $0.04 per million tokens. An earlier $0.03 reference was retired. v1.4.2.2, the same day, adds Imajev-4B at 67.37 and rank 1, on the unchanged v1.4.2 scorer. Prior scores stay, so Jev is still 63.29 and decider-4b v2 is still 64.13. A September 28 note says Imajev’s cost basis was corrected to one pinned server pass, and that no score, axis, rank, eligibility, or numeric value changed. Florian posted both names that evening and said a larger test set was still incoming.

The capability view we opened on September 28 averages Intelligence and Calibration and limits the list to systems within twice Jev’s cost and median latency. Imajev-4B leads that view at 66.3, from Intelligence 52.2 and Calibration 80.4, estimated $0.022 per 1,000, median 0.23 s, official rank 1. Jev is second there at 64.7, Intelligence 53.1, Calibration 76.3, $0.040, median 0.65 s, official rank 4. Plumb-4B is 64.2, official rank 2. decider-4b v2 is 62.2, official rank 3. The page says 52 Jev-class systems are in that list.

jevbench.app is a different project Yusuke posted on September 28. It replays recorded answers on small labeled sets. One of them, 21 examples, is printed as 20 of 21. It is not this board.

On September 28 at 17:49 UTC Florian posted v1.5. He wrote that Cygnet and Winnow-12B are close enough to share the top, and that Jev lands in third. The next post says the board scores Choice, Noul, and Score equally across 1,624 decisions per system, 904 open and 720 sealed, with half of Intelligence from each. The live page we opened on September 29 calls the release v1.5.0. Official score A is an equal-weight harmonic mean of Intelligence, Calibration, Speed, and Cost. An Intelligence, Speed, or Cost axis under 50 is multiplied by (axis / 50) squared. Of the 904 open decisions, 601 are published. The 720 sealed decisions are half of Intelligence. A gap penalty applies only above the field median gap, 5.2, plus 8. The page ranks 89 of 103 systems. It says 75 of 88 adjacent pairs are statistical ties, and it calls Cygnet and Winnow-12B Q8 joint leaders. Prices on this page are dollars per 1,000 decisions. We did not rerun the suite.

Cygnet is official rank 1 at 73.7. The axis table prints Intelligence 71.1, Calibration 87.0, Speed 91.0, Cost 56.4, and a gap of +2.7. Cost is about $0.028 per 1,000 decisions. The capability view lists capability 79.0 and a median latency of 0.23 s. The credit line says the shim is MIT and the frozen Gemma-4-12B-it weights are Apache-2.0 under Google’s Gemma policy. The repo is blockbrain-ai/cygnet-recipe.

Winnow-12B Q8 is official rank 2 at 73.2. The axis table prints 74.4, 84.1, 86.1, and 56.6, with a gap of -3.9 and about $0.028. Capability is 79.3. Median latency is 0.34 s. The credit line says Apache-2.0, including the Gemma 4 terms. The weights are EldanRing/Winnow-12B.

Jev 1.13.0 is official rank 3 at 72.1. Axes are 72.0, 88.0, 83.8, and 54.7. The gap is -0.9. Cost is about $0.032 per 1,000 decisions. Median latency on the hosted API is 0.62 s. Among 56 Jev-class systems, systems within twice Jev’s cost and median latency, the page gives Jev the highest capability, 80.0. The compare table on that page, Jev against Cygnet, prints chance-corrected competence of Choice 85.7 against 82.9 on the open set and 87.6 against 80.3 on the sealed set, Noul 47.8 against 57.9 and 48.6 against 55.7, and Score 81.2 against 76.6 and 81.1 against 73.2. The hard-tier cells in the same table are 53.8 against 58.0 open and 72.9 against 65.3 sealed.

Jev-Omni is official rank 4 at 71.5. decider-4b v2 is rank 5 at 71.3, with Intelligence 55.8, Calibration 85.6, Speed 90.9, Cost 64.5, a gap of +11.9, and about $0.015. SemIf is rank 6 at 68.7, Intelligence 51.3. spark-s1-4b-v6 is rank 7 at 68.2, Intelligence 62.1, Calibration 69.7, Speed 85.5, Cost 60.4, about $0.021, p50 0.41 s, capability 65.9. metask-jev-4b and Hopper are both 67.5, at ranks 8 and 9. Hopper’s Intelligence cell is 49.9, under the floor of 50, so that axis is scaled. Malkuth-4B is rank 10 at 66.8. djev is rank 13 at 64.2, Intelligence 72.3, Cost 48.2, about $0.053. classifier.dev scores 74.7 and is listed, not ranked. Its capability view is 82.5, Intelligence 75.8, Calibration 89.2, about $0.023, median 0.52 s.

Nine roster addenda use the same frozen sample and sit outside the ranked order. JevK5 v0.3 would place fourth at 71.9. Plumb-4B would place fourth at 71.6, interval 69.2 to 72.7, capability 71.6. Imajev-4B would place sixth at 70.4, interval 67.8 to 71.6, capability 70.8. Those point estimates are not the v1.4.2.2 scores. 67.37 and 63.29 stay on the older board. The page names METHOD-v1.5 and a pricing addendum. The hashes in the copy we saved were cut off, so this story does not print them. We did not rerun v1.5.

The same URL on September 30 calls the board v1.5.4 and numbers 106 systems. Cygnet, Winnow-12B Q8, and Jev 1.13.0 stay 73.7, 73.2, and 72.1. The page still calls Cygnet and Winnow joint leaders. The compact table rounds Jev’s axes to 72, 88, 84, and 55. The unrounded axes in the paragraph above stay 72.0, 88.0, 83.8, and 54.7. The capability control now says 60 Jev-class systems, and Jev’s 80.0 is still the first capability in that class.

Rows that were addenda on September 29 are numbered, and the page still tags them as roster addenda. JevK5 v0.3 is official rank 4 at 71.9 (A1). Plumb-4B is official rank 5 at 71.6 (A1). Jev-Omni, still 71.5, is rank 6. decider-4b v2, still 71.3, is rank 7, with Intelligence 55.8, a gap of +11.9, and about $0.015. Decision 4B v1.2 is rank 8 at 70.8 (A1). Imajev-4B is rank 9 at 70.4 (A2). Decision 4B v1.1 is rank 10 at 70.4 (A1). Manchego v2.1 is marked new, rank 11 at 68.8 (A5). SemIf, still 68.7, is rank 12. spark-s1-4b-v6, still 68.2, is rank 13. metask and Hopper, both still 67.5, are ranks 14 and 15. Malkuth-4B, still 66.8, is rank 16. Surogate Rune 26B-A4B v3 is rank 17 at 66.5 (A2). reflex 4B is rank 18 at 65.2. djev, still 64.2, is rank 20.

The tie line on that page is 69 of 77 adjacent pairs that have a published paired-bootstrap comparison. 28 adjacent pairs have no published comparison. The method line is still 904 open plus 720 sealed, 1,624 decisions, with sealed as half of Intelligence. Option A is still 25/25/25/25 with an Intelligence floor of 50. classifier.dev remains an honorable mention at 74.7, outside the numbered ranks. We did not rerun v1.5.4.

On October 1, 2026 we opened the same URL again. The page still says v1.5.4 and 106 ranked systems. A search of that page for “Mercury Decide” and for “Drex 1” returned no match. Inception’s post cites JevBench v1.4 and prints no score. Nace’s chart prints 199 of 231 against Jev at 201 of 231 on 231 public items. We did not rerun the board.

On October 4, 2026 we opened the same URL again. The page says v1.5.6 and numbers 110 ranked systems of 116. Cygnet, Winnow-12B Q8, and Jev 1.13.0 stay 73.7, 73.2, and 72.1. The page still calls Cygnet and Winnow joint leaders. imajev-4b is official rank 10 at 70.4, still tagged roster addendum A2, with axes 53.5, 88.1, 91.1, and 63.3, and an estimated $0.017 per 1,000 decisions. The September 30 rank for that row was 9. We did not rerun v1.5.6.

On October 7, 2026 we opened the same URL and the API board. Both pages name the headline release v1.6.1 and the board revision v1.7.12. The v1.7.12 note, dated that day, adds a language supplement of 354 items and a use-case supplement of 33 items. It says headline scores, Capability, Composite, and every rank are unchanged on both boards.

The open-weights board ranks systems the project ran itself. Jev 1.13.0 is an unranked reference row there. The capability list prints Intelligence 63.6, Calibration 90.6, capability 77.1, cost score 54.7, $0.032 per 1,000 decisions, and p50 0.24 s. The composite table on the same page prints 71.5, with rounded axes 64, 91, 91, and 55. Quyet-1.0-Large leads that capability list at 81.7, from Intelligence 73.4 and Calibration 90.0. v1.6.1 scores 1,500 decisions for a self-hosted system, 1,200 sealed and 300 public. v1.5 scored 1,624. 72.1 stays the v1.5.6 composite.

The API board ranks 23 hosted offerings. Sage 1.3.0 is composite rank 1 at 74.0. d1 is rank 2 at 73.0. Jev 1.13.0 is rank 3 at 71.5. wity-1 is rank 4 at 70.8. OpenAI Decisions, gpt-6-luna, is rank 5 at 62.5. The v1.7.10 note gives that row a 95% interval of 51.6 to 64.6 and capability 73.5. The capability list on the API board puts Sage first at 78.6 and Jev second at 77.1, p50 0.24 s. d1 is capability rank 3 at 74.4, Intelligence 62.1, Calibration 86.8, $0.017 per 1,000, p50 0.27 s, and official composite rank 2. The v1.7.8 note says d1 answered all 1,500 items on October 6, is scored without equating, and names a tariff of $0.04 per million input tokens. The v1.7.10 note says OpenAI Decisions answered 598 of 600 items on sealed set A5 plus the 300 public items, and that the row is equated. The list price in that note is $0.10 per million input tokens. Fastino GLiDE is composite rank 11 at 30.6. The composite row prints $0.137 per 1,000 decisions. The capability list prints 77.0 and places the row outside the Jev-class cost cap, at $0.14 per 1,000 decisions, 4.25 times the v1.5 Jev reference, p50 0.74 s. We did not rerun v1.6.1.

On the open-weights capability list, Imajev-4B is rank 17 at 61.9, Intelligence 34.6, Calibration 89.2, estimated $0.017 per 1,000, p50 0.29 s, official composite rank 26. Plumb-4B is capability rank 10 at 65.4, and official composite rank 10. Cygnet is capability rank 6 at 70.9, official composite rank 5. Winnow-12B Q8 is capability rank 5 at 71.2, official composite rank 3. 70.4 and 73.7 stay on the v1.5 board.

On October 8, 2026 we opened the same URL and the API board again. Both still name headline release v1.6.1. The board revision is v1.7.21, dated that day. The v1.7.21 note says more rows finished their L1, L2, and L3 supplement runs, and that this revision left headline scores, Capability, Composite, and every rank unchanged. The rank moves against the October 7 reading come from rows added in the revisions after v1.7.12. We reopened the published pages. We did not rerun the 1,500-item suite.

The API page ranks 26 hosted systems. The same page says every API offering measured is 30. Sage 1.3.0 is still first at 74.0. d1 is still second at 73.0. Mercury Decide is third at 72.4, linked as inception/mercury-decide:free on OpenRouter. Its capability card prints 73.6, Intelligence 65.9, Calibration 81.3, $0.018 per 1,000 decisions, and capability rank 5. Jev 1.13.0 is fourth at 71.5. The capability card still prints 77.1, Intelligence 63.6, Calibration 90.6, cost score 54.7, $0.032 per 1,000, p50 0.24 s, capability rank 2, and official composite 4. wity-1 is fifth at 70.8. Its capability card prints 79.1 and places the row outside the Jev-class cap because latency is 2.6 times the v1.5 Jev reference. SPX-CD Flash is sixth at 66.7. OpenAI Decisions is seventh at 62.5. Its capability card is still 73.5, Intelligence 56.9, Calibration 90.1, $0.052 per 1,000, p50 0.30 s, capability rank 6, official composite 7. The October 7 interval of 51.6 to 64.6 stays on the v1.7.10 note. Fastino GLiDE is fourteenth at 30.6. The composite row still prints $0.137 per 1,000. The capability card is still 77.0, outside the cost cap at $0.14 per 1,000, 4.25 times the v1.5 Jev reference, p50 0.74 s, and the rank line says official composite 14. Scores on the rows that were already printed on October 7 are the October 7 scores.

On the open-weights capability list the same day, Quyet-1.0-Large still leads at 81.7, and the rank line says official composite 3. Jev 1.13.0 is still the unranked reference. The capability card we read prints 77.1. H2O-Lightning-4B v1.1, a Qwen3.5-4B fine-tune named in the v1.7.14 note, is capability 75.0. That rank line says capability rank 5 and official composite 1. We did not copy its composite score. Winnow-12B Q8 is still 71.2, capability rank 13, official composite 7. Cygnet is still 70.9, capability rank 14, official composite 9. Plumb-4B is still 65.4, capability rank 23, official composite 25. Imajev-4B is still 61.9, Intelligence 34.6, Calibration 89.2, estimated $0.017 per 1,000, p50 0.29 s, capability rank 31, official composite 47. 70.4 and 73.7 stay on the v1.5 board. The v1.7.13 note says Surogate Rune 26B-A4B v3 and Xor 26B-A4B were added, and that no published score, axis, or cost changed. L3 is 1,118 items drawn on 7 Oct 2026. The v1.7.19 note says L3 items were reviewed by an OpenAI model, so L3 language and category cells on OpenAI rows carry that exposure. Headline scores do not use L3.

On October 11, 2026 we opened the API board again. It is still release v1.6.1, and it names board revision v1.7.44. The headline table ranks 27 hosted systems. Sage 1.3.0 is still first at 74.0. Its capability card prints 78.6, Intelligence 65.6, Calibration 91.6, an estimated $0.025 per 1,000 decisions, and median 0.15 s. d1 is still second at 73.0. Mercury Decide is still third at 72.4. Jev 1.13.0 is still fourth at 71.5, and the capability card is still 77.1. wity-1 is still fifth at 70.8. Microsoft-Decision-1 is sixth at 69.1. The row prints Intelligence 57, Calibration 84, Speed 82, Cost 61, and $0.019 per 1,000 decisions. The capability card prints 70.8, Intelligence 57.2, Calibration 84.4, and median 0.46 s. The v1.7.31 note, dated October 10, says that row is a native Azure Foundry run on 1,200 sealed items plus 300 public items, Capability 70.78, Calibration 84.40, Composite A 69.11, with 21 context-limit failures left in the denominator and a native median of 459 ms on 124 speed requests. SPX-CD Flash is seventh at 66.7. OpenAI Decisions is eighth at 62.5. The October 8 scores on the rows that were already printed stay the October 8 scores. H2O’s October 7 blog prints composite 72.5 for H2O-Lightning-4B v1.1 on revision v1.7.17. That 72.5 is the blog. The October 8 capability 75.0 stays the later card. We did not rerun the suite.

Opened October 17, 2026, about 01:48 UTC, the same API board is still release v1.6.1 and names board revision v1.7.46, updated 2026-10-16. The header prints 39 ranked API offerings. kettei-fat is first at composite 77.8, capability 86.3, estimated $0.0260 per 1,000, median 0.20 s. d1-ultra is second at 76.1. Sage 1.3.0 is third at 74.0. kettei-chub is fourth at 73.5. Jev 1.13.0 is seventh at 71.5, capability still 77.1, $0.0323 per 1,000, median 0.24 s. The kettei-fat row is a v1.6.27 draw measured 2026-10-16, put on the v1.6.1 scale. The write-up is kettei-fat. We did not rerun the suite.

Opened October 19, 2026, about 07:20 UTC, the JSON export is revision v1.6.29, latest_release v1.6.29, 184 ranked open-weight systems and 38 ranked API offerings. basal-omni-1.0-ultra is first of 184 on Capability at 82.54, Intelligence 74.06, Calibration 91.02. Open composite rank 3 at 71.76. H2O-Lightning-4B v1.1 is open composite rank 1 at 72.52. kettei-fat stays first of 38 hosted systems at composite 77.80, Capability 86.31. Jev 1.13.0 is seventh at 71.49, Capability 77.13. Jev 1.14.0 is ninth at 70.20, Capability 75.01, Intelligence 59.94, Calibration 90.08, p50 0.224 s. The ultra row is tagged measured_on 2026-10-17, measured_in v1.6.28, status anchor-runs-pending. We did not rerun the suite.

Quyet-1.0-Large’s homepage, opened October 12, 2026, prints a frozen v1.6.0 table: official score 71.39, rank 1 of 92, Capability 81.7, Intelligence 73.4, Calibration 90.0, and $0.045 per 1,000 decisions. Jev on that table is 71.09 and Capability 76.5. The homepage says that release scored 1,624 decisions. v1.6.1 on this page scores 1,500. The October 8 rank line still says official composite 3 at Capability 81.7. The weights are on the Quyet page. We did not reopen the frozen board file.