Published
Updated
A 400M encoder named Kev scores 79.96% against Jev on 17,476 of its own rows
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Sandipan Kundu posted skundu42/kev, a 400M encoder with Choice, Noul, and Score. On his held-out mixture Jev 1.13 scores 73.54% and this Kev scores 79.96%, ECE 0.015 against 0.098, median 132.7 ms against 388.4 ms. On a phishing set he did not retrain for, the same checkpoint flags 1,995 of 2,000 emails and accuracy is 50.25% against a published Jev figure of 62.6%. The root listing has no LICENSE file. This is not Jared Palmer's Kev. We did not rerun it.
Sandipan Kundu posted skundu42/kev on September 28. It is a 400M-parameter encoder: Choice over 2 to 16 candidates, a Noul, and a Score, with a 1,024-token cap on each prompt and candidate pair. Over-length requests are rejected. The repository root we listed has no LICENSE file. The Hugging Face weights are skundu42/kev. This project is not Jared Palmer’s Kev. The names collide. The repositories, base models, and tables do not. We did not train or call either one.
The paired table is in docs/benchmarks/README.md. Both sides saw the pinned test partition skundu42/kev-prepared at revision 5cea4539. Kev had trained and been calibrated on that mixture. Jev’s overlap with it is unknown. Jev was typesafe/jev-1.13-20260917 through OpenRouter. The checkpoint is aa9eb866, temperature 1.3734692667114208, on an Apple M3 Pro in FP32. A 100-row pilot, 10 examples from each source, seed 42, scores Kev 80.00% and Jev 81.00%. Medians are 129.7 ms and 390.4 ms.
The full run is 17,476 rows. Kev 79.96%, Jev 73.54%. That is 1,121 more accepted answers, 6.41 points. Brier is 0.276888 against 0.382336. ECE-15 on target mass is 0.015148 against 0.097899. Ordinal MAE on 997 scoring rows goes the other way: 0.392345 against 0.356360. Median latency is 132.7 ms against 388.4 ms. p95 is 607.3 ms against 499.5 ms, so the tail favors Jev. Of the pairs, both were right on 11,070 rows, only Kev on 2,903, only Jev on 1,782, and neither on 1,721. The recorded API bill is $0.343704, excluding warmups and unrecorded retries. Kev ran one request at a time. Jev used four HTTP workers. The clocks include Kev’s tokenization and Jev’s network time.
The lead sits in two sets. Jev is ahead on HellaSwag (93.45% to 86.15%, 2,000 rows), ANLI (74.35% to 58.10%), BoolQ (91.79% to 86.23%, 828 rows), and defeasible NLI (88.80% to 87.40%). Kev’s aggregate comes largely from FOL-NLI (83.20% to 60.30%) and zero-shot-label NLI (73.02% to 24.46%). The accuracy rule on this mixture accepts any candidate with positive gold mass. Jev’s probabilities were rounded to hundredths and renormalized. Raw log loss with a floor of 1e-300 is 9.942500 for Jev, because of those rounded zeroes. With a floor of 1e-6 it is 0.839409, against Kev at 0.474689.
An older paired report against Laya English, on an RTX 4090, is 79.93% against 51.63% on the same 17,476 rows. Laya’s native limits changed 1,597 inputs. On the 15,879 untouched rows the split is 79.44% against 52.93%.
The external suite uses other people’s prompts and does not call Jev again. On 2,000 PhishNChips emails the checkpoint’s accuracy is 50.25%, against a published Jev reference of 62.6%. AUROC is 0.34821 against 0.689. It marked 1,995 of 2,000 emails as phishing: 1,000 true positives, 995 false positives, 5 true negatives, 0 false negatives. Typed decisions, 400 cases and 2,000 questions, score 44.15% against a published Jev reference of 72.7%. Score MAE on 800 of those is 0.610459. Median latency is 123.4 ms per email and 2,008.5 ms per five-question case. The README says the mixture’s accuracy rule and the external exact-label rule are different tasks, and that the external Jev numbers are not a new paired run.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
skundu42/kev. The root file listing has no LICENSE file. Checkpoint aa9eb8668e2ab5675da1479993ae069c28124f10, temperature 1.3734692667114208. The paired Jev run used typesafe/jev-1.13-20260917 through OpenRouter.
Test partition skundu42/kev-prepared at 5cea4539b6676edfb4fd1fb60878e9aee602b0cd. Pilot, 100 rows, seed 42: Kev 80.00%, Jev 81.00%, median 129.7 ms and 390.4 ms.
Full run, 17,476 rows: Kev 79.96%, Jev 73.54%, Brier 0.276888 and 0.382336, ECE-15 0.015148 and 0.097899, ordinal MAE on 997 rows 0.392345 and 0.356360, median 132.7 ms and 388.4 ms, p95 607.3 ms and 499.5 ms. Both correct 11,070, only Kev 2,903, only Jev 1,782, neither 1,721. Reported API cost $0.343704.
Kev ran on an Apple M3 Pro. Jev used four HTTP workers. Laya English on the same 17,476, an older CUDA report: 79.93% against 51.63%. External Luni suite, Jev not called: PhishNChips 2,000 emails, accuracy 50.25% against a published Jev reference of 62.6%, AUROC 0.34821 against 0.689.
Kev flagged 1,995 of 2,000, with 995 false positives. Typed decisions, 400 cases and 2,000 decisions: 44.15% against a published 72.7%. Peak VRAM with BF16 weights on an RTX 4090 is 1.06 GiB in one configuration.
We did not rerun the calls.
Compare
Jared Palmer's jaredpalmer/kev is a different Apache-2.0 family on Qwen. On September 28 Christoph Görn scored that Kev-4B at 1 of 7 on eight of his own prompts. This 400M encoder's 79.96% is its training mixture, where any positive-target candidate counts as correct.
The phishing and typed numbers use exact labels, and the Jev cells there are published references.
Terms
- 79.96%
- skundu42/kev on 17,476 held-out rows from its own prepared split. Jev 1.13 on the same rows, through OpenRouter, is 73.54%. A 100-row pilot went the other way: 80.00% and 81.00%.
- 50.25%
- The same checkpoint on 2,000 PhishNChips emails, threshold 0.5, no fine-tune. It flagged 1,995 of 2,000. The published Jev reference on that set is 62.6%. Jev was not called again for this run.
- 1,995 of 2,000
- Emails the 400M model marked as phishing. True positives 1,000, false positives 995, true negatives 5, false negatives 0. AUROC 0.34821.