Published
Charly Poly put Jev next to open encoders, not GPT-5.6, and it lost on calibration
Browserbase's Charly Poly ran Banking77 against ModernBERT, BART-MNLI, and GLiClass. Jev won macro-F1 and automated about twice as much traffic at a 90% precision bar. Expected Calibration Error favored the 396M open encoder, and Jev assigned probability 0 to the right label 6.6% of the time.
Charly Poly, who works at Browserbase, posted on September 18 that comparing Jev with GPT-5.6 is the easy test, because Jev does not generate text. He ran Banking77 instead: 77-way intent routing with human labels, against three open-weight encoders that have done zero-shot classification for years.
Setup, as he wrote it: three seeds, thresholds fit on validation and reported on held-out test, the same label wording for every model. 24 of 24 runs finished with zero errors.
Macro-F1 he reports:
- Jev: 0.782 plus or minus 0.009
- ModernBERT-large-zeroshot: 0.712 plus or minus 0.020
- bart-large-mnli: 0.453 plus or minus 0.013
- gliclass-modern-base: 0.381 plus or minus 0.011
Jev wins. A 396M encoder you can run on a laptop lands within 7 points and costs nothing in API spend.
Calibration is the claim TypeSafe puts in the product name (RLCD, Reinforcement Learning for Calibrated Decisions). Expected Calibration Error, lower better: ModernBERT 0.081, Jev 0.105. Jev wins Brier (0.322 versus 0.422), which also rewards accuracy, so he treats that as the F1 win showing up again.
When each model said it was 80 to 95% sure, Jev was right 64% of the time and ModernBERT 80%. Overall he puts Jev about 10 points overconfident: 0.894 stated versus 0.794 actual.
Jev assigned probability exactly 0 to the correct label on 6.6% of decisions. None of the three open models did that once in 1,155 decisions each. A zero cannot be saved with a threshold, a fallback, or a re-rank.
The operational win is still Jev’s. Fit a cutoff on validation for 90% precision and apply it to the held-out set: Jev auto-handles 67.5% of traffic, ModernBERT 35.7%. About twice the automation at the same precision bar.
Someone objected that he compared a hosted API with a laptop. He ran bart-large-mnli on Hugging Face Inference as well. Same weights: 150 of 150 identical predictions. Hosted p50: Jev 299 ms, BART 5,131 ms, 17.2 times. That part of the speed claim held.
There is no gist or repository attached to the thread. We did not rerun Banking77.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The ten-post thread is the source. There is no public repository. Poly reports Banking77 (77-way intent, human labels), three seeds, thresholds fit on validation and reported on held-out test, identical label wording for every model, 24 of 24 runs with zero errors. Macro-F1: Jev 0.782 plus or minus 0.009, ModernBERT-large-zeroshot 0.712 plus or minus 0.020, bart-large-mnli 0.453 plus or minus 0.013, gliclass-modern-base 0.381 plus or minus 0.011. ECE (lower is better): ModernBERT 0.081, Jev 0.105. Brier: Jev 0.322 versus ModernBERT 0.422. In the 80 to 95% stated-confidence band, Jev was right 64% of the time and ModernBERT 80%. Overall he reports Jev overconfident by about 10 points (0.894 stated versus 0.794 actual). Jev assigned probability exactly 0 to the correct label on 6.6% of decisions; the three open models did that zero times in 1,155 decisions each. At a validation-fit 90% precision threshold, Jev auto-handled 67.5% of traffic and ModernBERT 35.7%. Hosted Hugging Face Inference BART, 150 of 150 identical predictions versus the laptop weights, p50 5,131 ms against Jev 299 ms (17.2x). We did not rerun the set. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; this story reports the thread as posted.
Compare
Most measurements on this desk put Jev next to a generative model. Aman Kumar's 16,000-call write-up also uses Banking77 (300 items) and finds Jev behind gpt-5.4-mini and gpt-5.6-luna there, which is the same 77-way set with a different comparator. jeff trails Jev on public labels and copies the HTTP API. This thread is the first encoder-only comparison we have with ECE, a zero-probability miss rate, and a precision-gated automation share. The missing repo is the limit: the numbers live in the posts.
Terms
- Expected Calibration Error
- A score for how far stated probabilities sit from observed accuracy. Lower is better. Poly reports 0.081 for ModernBERT and 0.105 for Jev on this Banking77 run.
- Zero-probability miss
- The model assigns probability 0 to the correct label. Poly reports this on 6.6% of Jev's decisions and on none of 1,155 decisions from each of the three open encoders. A zero cannot be recovered with a threshold.
- Precision-gated automation
- Fit a confidence cutoff on validation for a target precision, then apply it to held-out traffic. Poly's 90% precision bar auto-handled 67.5% of items with Jev and 35.7% with ModernBERT.