Jevre's test split is 0.9323, and held-out sources fall to 0.8366

Di Zhang posted on September 23 that trotsky1997/jevre can serve, evaluate, and train a local stand-in for Jev. The post says the training is RLCD with a parallel sampler. The README does not use those words. It describes a packed sequence, a tree attention mask so each candidate attends to the state, its own question, and its own tokens, a rank-512 readout, and a loss that combines Plackett-Luce ranking with a Brier term. The license is MIT. The checkpoint on Hugging Face is di-zhang-fdu/jevre, listed as 0.85B. The mixture is di-zhang-fdu/jevre-mix.

jevre serve speaks the TypeSafe Python SDK’s system_one API. Choice, Noul, and Score can share one request. The README’s example points TypeSafeClient at http://127.0.0.1:8000 with the API key local.

The released table is 12,000 evaluation rows, using the model’s raw probabilities. Test: accuracy 0.9323, ECE 0.0079, Spearman correlation of confidence and accuracy 0.976. Held-out sources: accuracy 0.8366, ECE 0.0944, the same Spearman 0.976. A shuffle that keeps the candidates and the label and replaces the state drops the test row to 0.4282 and the held-out row to 0.4369. The served HTTP path is accuracy 0.9267, 77 ms at the median, 109 ms at the 99th percentile. The published training run is one epoch, eight-way data parallel, 47 minutes. The train command takes the base model as $MODEL. The README does not name that base model in the section that describes the run.

A consistency suite checks candidate order, state padding, and distractor swaps. The API test compares served probabilities with in-process probabilities. We did not download the mixture or run either command.

Decider 1 publishes a Jev column on teacher-average labels: 0.768 against 0.727. Jevre’s 0.9323 has no Jev cell next to it. The number that moves when the sources change is the ECE, from 0.0079 on the test split to 0.0944 on held-out sources. The shuffle is what shows the state was carrying the test-split score.