Decision Index 0.2 prints Jev at 51.67, and a later post names another first place

apolinario/decision-index is a kit for rebuilding Decision Index, a benchmark for models that take a state and typed questions and return a probability for every option. The README on September 30, 2026 says the live board is Decision Index 0.2.1, dated September 27, and that the project is not affiliated with TypeSafe. Edition 0.2, dated September 24, is the earlier board. Code in the repository is MIT. The suite text comes from other datasets under their own terms, and the README says not to train on it.

Edition 0.2.1 rescores the same suite files. It counts 38 benchmarks. Arts and Human Taste is fixed at 10%, and the other four areas share the rest by the square root of how many benchmarks they hold. Gold benchmarks weigh 1.2 inside an area. RouterBench and SGD stay on the board and leave the index. The kit’s count for that edition is 120,340 requests, 119,898 that score after 442 exclusions, plus the same 30,419 for the seven benchmarks added in 0.2.

The index is the mean of five areas, times 100. Each area is the mean of its benchmarks. A benchmark is chance-corrected: 0 is random guessing and 100 is perfect, and a request that was not answered counts as wrong. MMLU left the index. Seven benchmarks were added, including MMLU-Pro, BBH, When2Call, RAGTruth, HoVer, New Yorker caption matching, and PhishNChips. The README’s count of the frozen rows is 121,057 requests after an ACOS subset, 120,615 that score after 442 exclusions, plus 30,419 requests for the new seven. Scores within 0.25 points share a rank.

tests/test_index02.py checks that math against published per-benchmark results. The four numbers named in the README are Jev 51.67, AutoJev-27B 50.94, Hopper 30.01, and Verdict 1.82. The README on September 30 says a rescore covers 64 entrants on the 0.2 board and 67 on the 0.2.1 board.

On September 30, 2026 we ran tests/test_index02.py and tests/test_index021.py from that day’s main. 13 tests passed in the first file and 19 in the second. The 0.2 locks are unchanged. The 0.2.1 test locks Jev at 57.89 in its fixture, with Rune 26B-A4B v3 at 57.44 and AutoJev-27B at 56.40, among 13 names. Drex is not one of those names.

On September 25, Florian S wrote that the Decision Index and JevBench measure different things. He called the index accuracy on 40 public benchmarks, and JevBench his own items plus a sealed set that also prices speed and cost. He wrote that Jev is first on raw intelligence on both, with JevBench Intelligence at 53.1 against 49.4, and that decider-4b v2 leads JevBench because it is faster and cheaper. He did not print 51.67.

The same afternoon, two posts said a model called Drex had taken first place: 51.73 against Jev 1.13.0 at 51.67, ahead on 23 of the 40 benchmarks, 70 tokens a decision against 367, and about 25 decisions a second on one H100. Movez also described a flap-or-glide game he marked as simulated, 20 seeds, 19 wins and one draw, about 136 ms against about 204 ms. The 51.67 matches the kit’s test. The 51.73 does not appear in that test, and we did not find Drex named in the README section we read.

The board we opened the same day does not name Drex either. data/index.json, generated 2026-09-28T00:39:36+00:00, is Decision Index 0.2.1. It lists 70 models plus Jev 1.13.0 at 57.91, then Surogate Rune 26B-A4B v3 at 57.44, Decider chat Gemma-4-31B at 57.33, and AutoJev-27B at 56.40. The 0.2 file from September 27 still has Jev at 51.67, and it has Decider chat Gemma-4-31B at 51.93, 0.26 above Jev. Feeding every per-benchmark skill in both files through the kit’s index02.aggregate reproduced the printed index within 0.01 on all 65 and 71 names. The largest gap was 0.0051.

Nace’s launch post prints the 51.73, and says Drex wins 22 of the 39 benchmarks scored for it. Its table lists Rune at 47.23 and Decider chat at 46.08, which match the September 27 file, and it does not list Decider chat Gemma-4-31B. The product page prints Drex 1.5 at 58.28 on Decision Index 0.2.1, first of 71, with Jev at 57.91 and a lead on 21 of 38. It says that row was scored on September 28 with the official 0.2.1 kit, and that the other cells are the public board’s scores from that day. 58.28 is not in the board file.

We sent one request through the kit’s http engine to https://api.typesafe.ai/v1/systemone, model jev-1.13.0. Warm-up was off, so this was the only request. The state was “The color is red.” The choice asked which color is named, red or blue. The noul asked whether the state names a color. The runner recorded status ok and named jev-1.13.0. The choice was red, with probabilities 1.0 and 0.0 and confidence 1.0. The noul was 0.94. Usage was 324 input tokens and 48 output tokens. Wall time was 646 ms. The runner finished at 2026-09-30T02:53:12Z.

We did not run the frozen suite. The repository does not ship the rows. Edition 0.2.1 scores 120,340 requests.

Maincode’s card on September 30 prints a self-run of this kit: Matilda 59.26 against the board’s Jev cell of 57.91. The card says the model is being submitted, so that 59.26 is not a row in the September 28 file. The area split, including Knowledge and Reasoning at 43.52 against Jev at 51.40, is on the Matilda page.

Nace’s September 30 post calls Drex 1.5 first on JevBench as well as on this index. The chart on the next post prints Jev 1.13.0 at 201 of 231 and Drex 1.5 at 199 of 231, with a hard-tier tie at 82 of 111. The product page prints a 128K context window and a size under 10 billion parameters. Those rows are on the Drex 1.5 page. Liquid’s chart the day before is a different reproduction, 58.9 against 57.9. We did not re-download the September 28 file, and we did not search it for the string d1. On October 1 a search of the JevBench page for “Drex 1” returned no match.

On October 1 Fastino’s blog printed a self-run of this same 0.2.1 scorer: GLiDE at 64.81 against the board’s Jev cell of 57.91. The blog says submissions for that edition are paused, so 64.81 is not a board row. On October 3 AutoTrust’s card printed another self-run, GEV-26B-Decide at an adaptive 62.48, with raw 70.66 and breadth 62.00. The card says the adaptive pass on Knowledge and Reasoning misses the kit’s latency cap. Both numbers are written up on their own pages, GLiDE and GEV-26B-Decide. We did not search the September 28 file for either name. The public kit still treats a gap of 0.25 as a tie, and both of these gaps are larger than that.

On October 3 StartLux’s README printed a further self-run: StartLux-Decision-27B at 63.88, the 35B-A3B at 61.55, and the 9B at 58.63, against the same 57.91. The 27B is ahead of that Jev cell on 31 of 38 benchmarks. The README says the runs are not on the board, and that 14 of the 38 benchmarks contributed their public train splits to training. Knowledge and Reasoning on the 27B is 44.3 against Jev at 51.4. The write-up is StartLux-Decision. We did not search the September 28 file for that name.

On October 5 the Decision-2.0-Vega-27B card printed another reproduction of the 0.2.1 kit, 56.5. The card does not print a Jev row. 56.5 is not in the September 28 file. The rest of that family is on the Decision 2.0 page. We did not search the September 28 file for Vega.

Opened October 8, 2026, the same space still says Decision Index 0.3. The header says 115 models, Jev jev-1.13.0, reproductions on one NVIDIA RTX PRO 6000, updated 2026-10-07. The lede says 114 open reproductions. The head-to-head panel still lists Perplexity Decider v1.1 at 62.8, Fastino GLiDE no-thinking at 60.2, and Torchcast Decision 27B at 59.9, with Jev as a reference at 60.1. The full-results table prints =2 on the GLiDE row, the Jev row, and the Torchcast row. We did not rerun the index.

Opened October 7, 2026, the same space headers edition v0.3, 112 models, Jev jev-1.13.0, reproductions on one NVIDIA RTX PRO 6000, updated 2026-10-06. The full-score summary, labelled all five areas, lists Jev at 60.1 as a reference. Rank 1 is Perplexity Decider v1.1 (27B) at 62.8. Rank 2 is Fastino GLiDE no-thinking (28B) at 60.2, and the button says open weights are coming soon. Rank 3 is Torchcast Decision 27B at 59.9. The v1.1 row then prints 45.8, 62.3, 64.7, 80.6, and 51.4, in the area order Knowledge, Language, Retrieval, Tools, Arts. It also prints an empty cell, 28B, 100.0%, and 104 ms. We did not keep the column titles for those three cells. Language lists Jev at 58.3 and v1.1 at 62.3. Retrieval lists Jev at 61.4 and v1.1 at 64.7. We did not copy a Jev reference for Knowledge, Tools, or Arts. The formula on the page is 0.20 times public, plus 0.50 times same-skill private, plus 0.30 times new domains. The public part is 100 times the weighted mean of the area scores, with GSM8K rebuilt and ForecastBench retired.

The v1.1 card prints its own table at 61.56 against Jev at 57.9. That table is a different reading from the space’s 62.8. Drex DLM prints 52.31 on a card labelled Decision Index 0.2, against the kit’s Jev cell of 51.67. A search of the 0.3 space for Drex returned no row. A search for “Decider v1 (27B)” returned no row. We did not rerun edition 0.3.