Published
Updated
The Jeeves README prints 0.865 on the public JevBench hard tier
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
PostHog/jeeves, MIT, is a Qwen3.5-9B LoRA plus a pointer head. The README's thinking run, greedy, 2,560-token cap, prints 0.865 on 111 public JevBench hard items, against a Kev-published Jev cell of 0.730. On the transfer overall, Jev is 0.800 and Jeeves is 0.746. The sealed judge tier is not in the JevBench rows. We did not run it.
PostHog/jeeves is on GitHub under an MIT badge, with weights at PostHog/jeeves. The README citation is Nicholas P. Waltz, 2026, “Jeeves: Reasoning Improves Jev-like Decisions.” The page says the project is inspired by Kev. K.Chatani posted a pointer on September 30 that restates the hard-tier pair 0.730 and 0.865. The post has no photo. We did not find an employer for Waltz on the pages we read.
The checkpoint is a Qwen3.5-9B LoRA, rank 16, on all projections, plus a pointer head, and a block-4 diffusion drafter. Training is supervised fine-tuning, then CISPO, stopped at step 402 of 624. The server speaks POST /v1/systemone and returns yes/no (noul), choice, and score. Think defaults to true. max_think is 2560.
The accuracy table is that thinking setup, greedy, with the 2,560-token cap. The README says the Kev-9B and Jev columns are the numbers Kev publishes, and that comparisons outside JevBench use different items from the same sources. Test overall, out-of-domain and held-out, item-weighted: Kev-9B 0.822, Jev 0.857, Jeeves 0.889. Transfer overall, MMLU-Pro and buried state: 0.579, 0.800, 0.746. On that transfer row the Jev column is higher.
The JevBench block is 231 public items. The README says the sealed judge tier is not included. The Kev column there is Kev-8B, starred because no Kev-9B JevBench number is published: overall 0.715, 0.866, 0.935. Hard, 111 public items: 0.451, 0.730, 0.865. Public ECE is Jev 0.049 and Jeeves 0.037. 0.730 times 111 is about 81, the count other notes print as 81 of 111. 0.866 times 231 is about 200. Jeeves prints 0.865 and 0.935, which are about 96 of 111 and about 216 of 231. The README does not print those counts.
The same column order on the other printed cells: QNLI 0.925, 0.925, 0.913. SciQ 0.963, 0.988, 0.991. TweetEval offensive 0.775, 0.813, 0.813. PAWS 0.763, 0.788, 0.875. MMLU 0.738, 0.900, 0.793. Emotion 0.600, 0.588, 0.647. Held-out rule structures 0.896, 0.885, 1.000. Contrastive policies 0.900, 0.963, 1.000. MMLU-Pro, 10-way: 0.515, 0.840, 0.739. Buried state 0.740, 0.700, 0.759. Unknowable items answered at probability at least 0.9, where lower is better: 0.000, 0.090, 0.055. On MMLU and MMLU-Pro the Jev column is higher.
A separate line says the same checkpoint without thinking scores 0.804 on “our test split (2,962 items),” against 0.840 with thinking. That 2,962-item split is not the 0.889 test-overall cell. On one H100 the page prints about 0.3 seconds without thinking, a 3.3 second median with it, and a p90 of 17 seconds for full chains. On 325 development questions the full thinking run is 0.825, mean 1,138 reasoning tokens, median 3.3 seconds, p90 17.1 seconds. Capping max_think at 768 and setting nothink_threshold to 0.9 is 0.806, 344 tokens, 2.0 seconds and 5.6 seconds. No thinking on that set is 0.775 at about 0.3 seconds.
The example curl prints a department choice of billing with confidence 0.19. The probabilities on that example are returns 0.4, shipping 0.14, and billing 0.46. The page names Python 3.12, CUDA, and Hopper for the FP8 kernel. Drafter throughput is 109 tokens a second plain, 176 with block-4 (printed as 1.6 times), 193 with block-8, and about 960 tokens a second in total for block-4 across eight questions. We did not run it.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The X post is @chataclaw, September 30, 2026, 03:33 UTC. It has no attached photo. It restates the hard-tier move from 0.730 to 0.865, which matches the README table. The README citation is Nicholas P. Waltz, 2026, "Jeeves: Reasoning Improves Jev-like Decisions." The badge is MIT. The page says the work is inspired by Kev. We did not find an employer for Waltz on the pages we read.
The checkpoint is a Qwen3.5-9B LoRA, rank 16, on all projections, plus a pointer head, and a block-4 diffusion drafter. Training is SFT, then CISPO, stopped at step 402 of 624. The server is POST /v1/systemone, with yes/no (noul), choice, and score. Think defaults to true. max_think is 2560.
Accuracy rows below are with thinking, greedy, and that 2,560-token cap. The README says the Kev-9B and Jev columns are the numbers Kev publishes, and that Kev and Jev comparisons outside JevBench use different items from the same sources. The JevBench block is 231 public items. The README says the sealed judge tier is not included. The Kev JevBench column is Kev-8B, with a star that no Kev-9B JevBench number is published.
Test overall, out-of-domain and held-out, item-weighted: Kev-9B 0.822, Jev 0.857, Jeeves 0.889. Transfer overall, MMLU-Pro and buried state: 0.579, 0.800, 0.746. JevBench overall: 0.715, 0.866, 0.935. JevBench hard, 111 public items: 0.451, 0.730, 0.865. Public ECE: Jev 0.049, Jeeves 0.037.
Other README cells, same column order: QNLI 0.925, 0.925, 0.913. SciQ 0.963, 0.988, 0.991. TweetEval offensive 0.775, 0.813, 0.813. PAWS 0.763, 0.788, 0.875. MMLU 0.738, 0.900, 0.793. Emotion 0.600, 0.588, 0.647. Held-out rule structures 0.896, 0.885, 1.000. Contrastive policies 0.900, 0.963, 1.000. MMLU-Pro 10-way 0.515, 0.840, 0.739. Buried state 0.740, 0.700, 0.759. Unknowable items answered at probability at least 0.9, lower better: 0.000, 0.090, 0.055.
A separate line says the same checkpoint without thinking is 0.804 on "our test split (2,962 items)," and 0.840 with thinking. That split is not the 0.889 test-overall cell. On one H100 the README prints about 0.3 seconds without thinking and a 3.3 second median with it, with p90 of full chains at 17 seconds. On 325 development questions: full thinking is 0.825, mean 1,138 reasoning tokens, median 3.3 seconds, p90 17.1 seconds. max_think 768 plus nothink_threshold 0.9 is 0.806, 344 tokens, 2.0 seconds and 5.6 seconds. No thinking is 0.775 at about 0.3 seconds.
The example curl prints a department choice of billing at confidence 0.19. The probabilities on that example are returns 0.4, shipping 0.14, billing 0.46. Python 3.12 and CUDA are named, and Hopper for the FP8 kernel. Drafter throughput: plain 109 tokens a second, block-4 176 (1.6 times), block-8 193, and block-4 across eight questions about 960 tokens a second in total. We did not run it.
Compare
0.730 times 111 is about 81, the public-hard count other notes print as 81 of 111. 0.866 times 231 is about 200, the count Open-Jev prints as 200 of 231. Jeeves prints 0.865 and 0.935. 0.865 times 111 is about 96. 0.935 times 231 is about 216. The README does not print those integer counts.
JevBench v1.5.4 is a different scale. It includes a sealed judge tier, which this README says is left out, and it prices speed and cost. Jev 1.13.0 there is 72.1. On the transfer overall and on MMLU and MMLU-Pro in this table, the Jev column is higher than Jeeves.
Terms
- 0.865
- Jeeves on the README's public JevBench hard tier, 111 items, with thinking. The Kev-published Jev cell on that row is 0.730. Kev-8B is 0.451. The sealed judge tier is not in this row.
- 0.889
- Jeeves on the README's test overall, out-of-domain and held-out, item-weighted, with thinking. Kev-9B is 0.822 and the Kev-published Jev cell is 0.857. The 2,962-item split without thinking, 0.804, is a different line.
- 0.746
- Jeeves on the README's transfer overall, MMLU-Pro and buried state. Kev-9B is 0.579. The Kev-published Jev cell is 0.800.