A 37-dataset paper scores frozen jev-1.13.0 against two open models

This topic was created 20 days ago, and the information it contains may have evolved or changed since then.

Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa submitted arXiv:2609.37647 on September 29, 2026. The abstract says jev-1.13.0, one frozen template per dataset, made 346,009 requests for under $10. Jev beats Qwen3.8-27B on 27 of 37 datasets and Gemma-4-E4B on all 37. This note uses the abstract. The HTML and the PDF were not opened.

Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa submitted arXiv:2609.37647 on September 29, 2026. The title is “Evaluating and Benchmarking the System One Model Jev.” This note follows the abstract. The HTML and the PDF were not opened, so no table from those files is in here.

The setup is jev-1.13.0, zero-shot, one frozen template per dataset, on the full evaluation splits. The abstract counts 37 datasets and 346,009 requests, for under $10. The datasets cover classification, routing, natural-language inference, reading comprehension, commonsense, moderation, legal clause analysis, and rubric scoring. Qwen3.8-27B and Gemma-4-E4B are scored on the same requests. Their score is the exact next-token probability of each option.

What the abstract prints

Jev is in the 95% to 99% band on IMDB, SST-2, HellaSwag, and ARC. On Belebele, across 122 languages, it is 86.7%. It beats Qwen3.8-27B on 27 of the 37 datasets. Qwen leads the other nine, and the abstract says none of those nine leads sits outside the bootstrap intervals. Jev beats Gemma-4-E4B on all 37. All three models get worse on low-resource languages, on fine-grained or noisy labels, and on rubric scores.

Choice probabilities are described as well calibrated, and as usable for selective prediction, where a low-confidence call can be refused. Binary probabilities rank the items well and land poorly against a fixed threshold of 0.5. Thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. That 0.75 is the tuned line. It is not the zero-shot accuracy.

Calculation-heavy MMLU questions are 94%, against 91% on the other MMLU questions. The abstract says both open models find the calculation items harder, and that all three models do on C-Eval. Rotating the options does not change Jev’s accuracy. Removing the question drops it near chance. The authors read that pair as evidence against a shallow memorization of option order, and they say it leaves memorized question-answer pairs possible.

The abstract names the code at AppliedMachineLearning-Lab/jev-benchmarking and the responses at Zenodo 10.5281/zenodo.23039006. We did not open the repository.

The 86.7% Belebele line is 122 languages in this abstract. The Perplexity panel prints Belebele at 95.00% for Jev on 500 rows. Those two figures are different slices. We did not rerun the 37 datasets.

This site's reading

Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.

Verify

arXiv:2609.37647, "Evaluating and Benchmarking the System One Model Jev." Authors Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa. Submitted September 29, 2026, 14:15:04 UTC. Version 1 only, as of this reading. We opened the abstract page. We did not open the HTML body or the PDF, so this note does not add table rows from either file.

The abstract says jev-1.13.0, zero-shot, one frozen template per dataset, on the full evaluation splits. The set is 37 datasets: classification, routing, NLI, reading comprehension, commonsense, moderation, legal clause analysis, and rubric scoring. The request count is 346,009, for under $10. Qwen3.8-27B and Gemma-4-E4B are scored on the same requests by exact next-token probabilities over the options.

Printed ranges and counts in the abstract: 95% to 99% on IMDB, SST-2, HellaSwag, and ARC. 86.7% on Belebele across 122 languages. Jev beats Qwen on 27 of 37 datasets. None of Qwen's nine leads falls outside the bootstrap intervals. Jev beats Gemma on all 37. All three models degrade on low-resource languages, on fine-grained or noisy labels, and on rubric-based quality judgments.

Choice probabilities are described as well calibrated and usable for selective prediction. Binary probabilities rank well and sit poorly against a fixed 0.5 threshold. Thresholds tuned on training data raise micro-F1 on UNFAIR-ToS from 0.50 to 0.75. Calculation-heavy MMLU questions are 94%, against 91% on the other MMLU questions. Both open models, and all three models on C-Eval, find the calculation items harder.

Rotating the options leaves Jev's accuracy unchanged. Withholding the question drops it near chance. The abstract says that rules out shallow memorization of option order, and does not rule out memorized question-answer pairs. The code URL in the abstract is github.com/AppliedMachineLearning-Lab/jev-benchmarking. Responses are at doi.org/10.5281/zenodo.23039006. We did not open the repository README.

Compare

Belebele at 86.7% across 122 languages is this abstract. Perplexity's October 1 chart prints Belebele at 95.00% for Jev on 500 rows. Those are different slices. The 346,009 requests are the abstract's count for this paper. A later post that restates the count and adds a second item total is not used here, because that second total is not in the abstract.

UNFAIR-ToS moving from micro-F1 0.50 to 0.75 is a threshold fit on training data. It is not the zero-shot accuracy. Qwen's nine leads stay inside the bootstrap intervals the abstract cites. The abstract does not print those nine dataset names.

The reference models are scored by next-token probabilities over the options, on the same requests. That is a different protocol from a decision head. We did not rerun the 37 datasets.

Terms

frozen template: The abstract's setup for jev-1.13.0. One prompt template per dataset, held fixed, zero-shot, on the full evaluation split. The open reference models see the same requests and are scored by exact next-token probabilities over the options.
0.50: The abstract's micro-F1 on UNFAIR-ToS before thresholds are tuned on training data. After that tuning the same line is 0.75. Binary probabilities are described as ranking well and sitting poorly against a fixed 0.5 cutoff.