Published
Updated
hotdog-27B leads a 1,631-decision holdout and trails on 74 public items
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Michael posted hotdog-27B, a rank-16 LoRA on Qwen3.8-27B that reads two logits and returns yes or no. On an independent holdout it is 1,603/1,631 against Jev 1.13.0 at 1,587/1,631. On 74 native-binary public JevBench items it is 60/74 against 62/74. The second interval includes zero.
Michael posted hotdog-27B on September 22. The chart calls it an open-weight binary classifier, 27B dense, Qwen3.8, a Tinker LoRA, and it draws two bars: an independent holdout of 1,631 decisions, and a public JevBench subset of 74. The model card on opensporks/hotdog-27B prints the unrounded counts. Training code is mrmps/tinker-binary, Apache-2.0. The adapter is Apache-2.0 as well.
The model takes text and a yes/no question. It reads the logits for A (false) and B (true) at the same token position and does not generate a reason. The released file is a rank-16 LoRA in Tinker format, about 483 MB, not a PEFT adapter a stock loader can apply. The card says the evaluated deployment used an H200. Temperature is fixed at 1.0 and the decision threshold at 0.5. Inputs longer than 8,192 tokens are rejected.
On the independent test, 1,631 decisions in 1,002 groups, hotdog-27B is 1,603/1,631 (98.28%) and Jev 1.13.0 is 1,587/1,631 (97.30%). The paired difference is +0.98 points. The 95% interval, from 10,000 cluster bootstrap replicates that keep related decisions together, is +0.19 to +1.79. Scalar Brier is 0.013175 against 0.026884. Expected calibration error, ten equal-width bins, is 0.58% against 6.40%. The card says the groups are disjoint from training and that the test still shares source datasets with the training mix.
On the public text JevBench slice, 74 native-binary decisions in 62 groups, hotdog-27B is 60/74 (81.08%) and Jev is 62/74 (83.78%). The difference is -2.70 points. The interval is -12.20 to +5.63, so it includes a tie and a lead either way. Brier is 0.125090 against 0.106305. ECE is 11.95% against 6.54%. The card says this row does not establish a JevBench win: it is not the 534-task composite, not the private items, and not the interactive bench at jevbench.dev. Both evaluations used one frozen model, with complete coverage and no later tuning, dated September 22, 2026. Latency is not compared, because the serving transports differ.
The selected checkpoint is step 96 of a low-rate continuation. The card says that ancestry consumed 13,017 distinct rows and 16,384 exposures, chosen by lowest development Brier among candidates within one point of the best development accuracy. The development set had 1,468 decisions. No fitted calibration was applied. JevBench examples were kept out of training and out of selection. The post says specialized binary models could later be served on classifier.dev. The card does not say this adapter is that host’s current fast tier. On the JevBench page this desk already read, that fast tier is Jev behind another API.
We did not load the adapter or call Jev.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @michael_chomsky, September 22, 2026, 19:10 UTC. The chart on that post rounds the same comparison: independent holdout 98.3% versus 97.3% on 1,631 decisions, public JevBench subset 81.1% versus 83.8% on 74 decisions, paired gaps +0.98 and -2.70 points. The model card we read gives the fractions.
Independent test, 1,631 decisions in 1,002 groups: hotdog-27B 1,603/1,631 (98.28%), Jev 1.13.0 1,587/1,631 (97.30%), difference +0.98 points, paired 95% interval +0.19 to +1.79. Public text JevBench, 74 native-binary decisions in 62 groups: 60/74 (81.08%) versus 62/74 (83.78%), -2.70 points, interval -12.20 to +5.63.
Scalar Brier: independent 0.013175 versus 0.026884; public 0.125090 versus 0.106305. ECE, ten equal-width bins: independent 0.58% versus 6.40%; public 11.95% versus 6.54%.
The card says both sets were one frozen model, complete coverage, no later tuning, September 22, 2026, and that the public row does not establish a JevBench win. Intervals are 10,000 paired cluster bootstrap replicates. The independent test uses disjoint document groups and shares source datasets with training.
Latency is not compared. Backbone is dense Qwen3.8-27B. The adapter is rank 16, about 483 MB, Apache-2.0, Tinker format. Selected checkpoint is step 96 of a low-rate continuation. Ancestry consumed 13,017 distinct rows and 16,384 exposures.
Development set 1,468 decisions. Temperature 1.0, threshold 0.5. Inputs longer than 8,192 tokens are rejected. The evaluated deployment used an H200. Training code is mrmps/tinker-binary, Apache-2.0.
Raw training rows are not in the release. JevBench examples were evaluation-only. The post says binary models could later be served on classifier.dev. The card does not say this adapter is that host's fast tier.
We did not load the weights or call Jev.
Compare
JevBench's composite is 534 decisions. This public row is 74 native-binary items from the public text repository. Open-Jev scored 231 public tasks and put Jev at 200/231. SimpleJev's Qwen3.8-27B row on the composite is 67.3, a different readout and a different score.
classifier.dev's fast tier, on the JevBench page this desk already read, is hosted Jev behind another API, not this LoRA.
Terms
- hotdog-27B
- A rank-16 LoRA on dense Qwen3.8-27B. It reads the logits for A (false) and B (true) at one token position and returns a probability. Apache-2.0 adapter, Tinker format, selected at temperature 1.0.
- native-binary JevBench
- The 74 public text items in this card that are already yes/no decisions. The card says they are not the 534-task composite, the private tests, or the interactive bench at jevbench.dev.