Open-Jev 27B v1.1 answers 197 of 231 public JevBench tasks

Zefan Cai posted on September 21 that all five runs of a public JevBench slice had finished. He is a PhD student at UW-Madison. The project is Open-Jev: LoRA adapters and a scalar decision head on Qwen3.5-2B and Qwen3.5-9B. The checkpoints return a probability over the candidates you supply. They are an independent implementation. The README says they do not reproduce TypeSafe’s training method or weights.

The name collides with other projects. SemIf used to be called OpenJev and reads option logits from a frozen Qwen3.5-4B. The audit file also separates this repo from Kotoba and Codiv projects that use the same name. Source is MIT. The adapters are Apache 2.0 and still need the pinned Qwen base weights.

The audited file covers 231 public tasks out of 534. That is 72 original, 48 easy, and 111 hard: 74 Noul, 139 Choice, and 18 Score. The other 303 tasks are unavailable, and the file does not compute a full-benchmark composite. Upstream commit is f8ce713. The JSON was generated at 06:27 UTC on September 21. The publication gate says native replay, hosted aggregate replay, and hosted raw-response replay agree.

Correct answers on those 231 tasks: released 2B 150 (64.94%), released 9B 179 (77.49%), Jev 1.13.0 200 (86.58%), GPT-5.6 Luna 206 (89.18%), GPT-6 Astra 231 (100%). The easy tier is 48/48 for every model. Original tier: 56/72, 65/72, 71/72, 69/72, 72/72. Hard tier: 46/111, 66/111, 81/111, 89/111, 111/111.

Timing is one observation per task, concurrency 1, no warmups and no retries. P50 full-response wall time is 138 ms for 2B, 189 ms for 9B, 291 ms for Jev, 954 ms for Luna, and 2,206 ms for Astra. Open-Jev used one H100 80GB, loopback HTTP, prefix cache off, candidate batch size 1. Hosted models used a fresh HTTPS connection, so the clock includes transit. The file treats that gap as a deployment measurement.

GPT answers are verbalized probabilities in constrained JSON, with reasoning set to none for Luna and low for Astra. Open-Jev and Jev return native probabilities. Candidate order differs on 119 of 139 Choice tasks, and the run does not control for that. Jev had 230 of 231 probability vectors inside the strict 0.001 sum tolerance. One vector was normalized and still scored. The other four models were strict-valid on all 231. Derived usage-times-tariff estimates are about $0.038 per 1,000 decisions for Jev, $0.179 for Luna, and $8.82 for Astra. The file labels them as estimates. Local GPU time is unmetered.

On the 15 public temporal and numeric tasks, the correct counts are 1, 2, 4, 5, and 15 for 2B, 9B, Jev, Luna, and Astra. Jev’s long-policy family is 12/19 against Luna 16/19 and Astra 19/19.

A separate holdout on the synthetic release-v2 labels, 10,046 hard test rows, is 94.71% for 2B and 97.54% for 9B. Those rows are not the JevBench tasks.

On September 23 Cai posted Open-Jev 27B v1.1. The package is a LoRA adapter and a scalar head on Qwen3.8-27B, pinned to revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. The card counts 15,471,617 trained parameters. Base weights are downloaded separately. The adapter and head are Apache 2.0. The loader stays MIT. The server line is still python -m jev.server on port 8791, and an input over 4,096 tokens per candidate is rejected.

This training stage used 148,639 rows and warm-started from an earlier 27B checkpoint at optimizer step 26,500. LoRA rank is 8 and alpha is 16. The saved temperature is 2.5343690298472983, fitted on 512 calibration rows. The public dataset has 326,619 records and 147,139 training rows. It omits 2,053 Wikispeedia rows, 1,500 of them from training, because redistribution of that graph was not confirmed. The card says those public files are not the bytes this checkpoint trained on. The internal scores include the omitted rows.

The internal audit covers 127,787 rows: 43,301 test and 84,486 out of distribution. It reports zero missing, duplicate, or failed predictions. Hard-label accuracy on the older panels is 9,876/10,046 (98.31%) and 14,825/15,446 (95.98%). On the expanded panels it is 41,357/42,789 (96.65%) and 80,934/83,924 (96.44%). Those expanded panels include 1,074 soft-label rows, which stay in the probability metrics and stay out of the hard-label rates. The older panels are subsets of the expanded ones. The run used the saved temperature, a 4,096-token limit, BF16 base weights, an FP32 head, and no prefix cache.

The JevBench number uses another protocol: the same 231 public tasks, a 16,384-token limit, and no prefix cache. Open-Jev 27B v1.1 is 197/231 (85.28%) and 80/111 on the hard tier (72.07%). Jev 1.13.0 on that card is 200/231 and 81/111. The 2B and 9B rows are the checkpoints from September 21, 150 and 179. Cai says new 2B training has stopped and new 9B training has not started. The card prints no latency for 27B and says a shared-node training clock is not an HTTP speedup. A lexical screen found no overlap with 531 frozen visible benchmark requests. The card says that screen does not rule out paraphrases or pretraining exposure.

We did not rerun either protocol. TypeSafe’s Master Customer Agreement section 2.3(f) forbids customers from publishing benchmarks of the Services. The figures above are the ones in the published JSON and on the model card.

On October 2 Cai posted one more run, a 64-step pilot on the released 2B checkpoint. The note is frontier-controls-v4.md. It uses 256 training rows, 64 calibration rows, and 128 rows each for test and out of distribution. The backbone stays frozen. A LoRA and the head train on one H200, with the same 4,096-token limit. The same records are scored before and after.

Test accuracy goes from 84/128 to 99/128, 65.6% to 77.3%. Out of distribution goes from 83/128 to 95/128, 64.8% to 74.2%. Test NLL goes from 0.8900 to 0.6046, Brier from 0.4903 to 0.3266, and 15-bin ECE from 0.1353 to 0.1088. Out of distribution, NLL goes from 0.9080 to 0.6998, Brier from 0.5028 to 0.3733, and ECE from 0.1736 to 0.1261. On the 32 noul rows, test goes from 24/32 to 26/32 and out of distribution from 22/32 to 26/32.

Families of 20 or 24 items do not all move the same way. Authorization goes from 75% to 100% on both slices. Negation goes from 95% to 100% on the test slice and from 85% to 100% out of distribution. Irrelevant policies go from 75% to 100% on both. Numeric candidates stay at 30% in distribution and fall from 35% to 30% out of distribution. Timeline goes from 58.3% to 62.5% on the test slice and from 58.3% to 54.2% out of distribution. Account state goes from 62.5% to 75% on the test slice and from 62.5% to 66.7% out of distribution.

The saved temperature moves from 1.518796342858676 to 1.5446931834967927. Hard accuracy is an argmax, so that temperature change does not move the correct counts. NLL, Brier, and ECE include the new weights and the new temperature together. The training commit is 80ca8e81d08992cb2a4cbb6a0caa1355ad3e5aee. The runner took 185.99 seconds. Reloading the checkpoint, the max error on three probes is 0.0. The note calls the rows synthetic controls. They are not the 231 public JevBench tasks, and they are not BANKING77.

We did not rerun the pilot.