Published
Open-Jev's released 9B checkpoint answers 179 of 231 public JevBench tasks
Zefan Cai published an audited run of the already released Open-Jev 2B and 9B checkpoints on the 231 public JevBench tasks. The 9B model is correct on 179. Hosted Jev 1.13.0 is correct on 200, GPT-5.6 Luna on 206, and GPT-6 Astra on all 231. The other 303 tasks stay private, so this is not a full JevBench score.
Zefan Cai posted on September 21 that all five runs of a public JevBench slice had finished. He is a PhD student at UW-Madison. The project is Open-Jev: LoRA adapters and a scalar decision head on Qwen3.5-2B and Qwen3.5-9B. The checkpoints return a probability over the candidates you supply. They are an independent implementation. The README says they do not reproduce TypeSafe’s training method or weights.
The name collides with other projects. SemIf used to be called OpenJev and reads option logits from a frozen Qwen3.5-4B. The audit file also separates this repo from Kotoba and Codiv projects that use the same name. Source is MIT. The adapters are Apache 2.0 and still need the pinned Qwen base weights.
The audited file covers 231 public tasks out of 534. That is 72 original, 48 easy, and 111 hard: 74 Noul, 139 Choice, and 18 Score. The other 303 tasks are unavailable, and the file does not compute a full-benchmark composite. Upstream commit is f8ce713. The JSON was generated at 06:27 UTC on September 21. The publication gate says native replay, hosted aggregate replay, and hosted raw-response replay agree.
Correct answers on those 231 tasks: released 2B 150 (64.94%), released 9B 179 (77.49%), Jev 1.13.0 200 (86.58%), GPT-5.6 Luna 206 (89.18%), GPT-6 Astra 231 (100%). The easy tier is 48/48 for every model. Original tier: 56/72, 65/72, 71/72, 69/72, 72/72. Hard tier: 46/111, 66/111, 81/111, 89/111, 111/111.
Timing is one observation per task, concurrency 1, no warmups and no retries. P50 full-response wall time is 138 ms for 2B, 189 ms for 9B, 291 ms for Jev, 954 ms for Luna, and 2,206 ms for Astra. Open-Jev used one H100 80GB, loopback HTTP, prefix cache off, candidate batch size 1. Hosted models used a fresh HTTPS connection, so the clock includes transit. The file treats that gap as a deployment measurement.
GPT answers are verbalized probabilities in constrained JSON, with reasoning set to none for Luna and low for Astra. Open-Jev and Jev return native probabilities. Candidate order differs on 119 of 139 Choice tasks, and the run does not control for that. Jev had 230 of 231 probability vectors inside the strict 0.001 sum tolerance. One vector was normalized and still scored. The other four models were strict-valid on all 231. Derived usage-times-tariff estimates are about $0.038 per 1,000 decisions for Jev, $0.179 for Luna, and $8.82 for Astra. The file labels them as estimates. Local GPU time is unmetered.
On the 15 public temporal and numeric tasks, the correct counts are 1, 2, 4, 5, and 15 for 2B, 9B, Jev, Luna, and Astra. Jev’s long-policy family is 12/19 against Luna 16/19 and Astra 19/19.
A separate holdout on the synthetic release-v2 labels, 10,046 hard test rows, is 94.71% for 2B and 97.54% for 9B. Those rows are not the JevBench tasks. A 27B run had started optimizer updates by 07:31 UTC on September 21. Its final scores are still pending. The local server is python -m jev.server on port 8791.
We did not rerun the 231 tasks. TypeSafe’s Master Customer Agreement section 2.3(f) forbids customers from publishing benchmarks of the Services. The figures above are the ones in the published JSON.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 21 post from @Zefan_Cai is the primary source. The audited file is jevbench.json, generated 2026-09-21T06:27:47Z, status complete, publication gate passed. Scope is 231 public tasks out of 534: original 72, easy 48, hard 111. Types: 74 Noul, 139 Choice, 18 Score. Upstream JevBench commit f8ce713. Open-Jev here is a Qwen LoRA plus a scalar decision head. The file says it is a different system from the Kotoba and Codiv projects that also use the name OpenJev. SemIf, formerly called OpenJev, is a separate logit readout. Correct counts: released 2B 150/231 (64.94%), released 9B 179/231 (77.49%), Jev 1.13.0 200/231 (86.58%), GPT-5.6 Luna 206/231 (89.18%), GPT-6 Astra 231/231 (100%). Hard tier: 46/111, 66/111, 81/111, 89/111, 111/111. Easy is 48/48 for all five. P50 full-response wall time: 138 ms, 189 ms, 291 ms, 954 ms, 2,206 ms. Jev strict-valid 230/231 with one vector renormalized. Derived tariff estimates per 1,000 decisions: Jev about $0.038, Luna about $0.179, Astra about $8.82. Local compute is unmetered. Open-Jev ran on one H100 with loopback HTTP and prefix cache off. Hosted calls are fresh HTTPS. GPT returns verbalized probabilities, reasoning none for Luna and low for Astra. Candidate order differs on 119 of 139 Choice tasks. One observation per task, no warmups, no retries. On the 15 temporal/numeric tasks the counts are 1, 2, 4, 5, and 15. We did not rerun the suite. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; the figures are reported as published.
Compare
JevBench's composite on the full 534 tasks is a different number. The table this desk read as v1.2.8 puts Jev at 75.4 and SemIf at 74.7, with Luna's hard tier at 94.5% and its composite at 66.2 because of price. This file scores only the 231 public tasks and does not compute that composite. Kev is another Qwen LoRA with a pointer head. Its JevBench rows are separate checkpoints. Open-Jev's own release-v2 holdout, 10,046 hard test labels, is 94.71% for 2B and 97.54% for 9B against synthetic references, which is not this public subset.
Terms
- Open-Jev
- Zefan Cai's Qwen3.5 LoRA adapters plus a trained decision head. The 2B and 9B checkpoints return Choice, Noul, and Score probabilities. A different project from SemIf, which previously used the name OpenJev.
- JevBench public subset
- The 231 public tasks in JevBench, out of 534. Original 72, easy 48, hard 111. The other 303 tasks are private or judge-only, so a score on this slice is not the full benchmark composite.