Published
Aman Kumar ran about 16,000 Jev calls and called it a filter, not a classifier
Aman Kumar compared Jev with gpt-5.4-mini and gpt-5.6-luna on four public sets, then on production page gates and an inbox. On short text it tied or led. On a whole-page read it fell apart. He published the public cases and per-item answers.
Aman Kumar posted on September 18: is Jev a classifier or a filter? About 16,000 calls later he answers filter. The write-up is on amankumar.ai. Public cases and per-item answers are in onlyoneaman/jev-eval.
He compared Jev with the small, fast tier most pipelines already use. Four public sets, 300 random items each, same question and the same one-line label text. gpt-5.4-mini and gpt-5.6-luna answered through the Responses API with reasoning set to low.
Accuracy he reports:
| dataset | Jev | gpt-5.4-mini | gpt-5.6-luna | Jev right when confident (share) |
|---|---|---|---|---|
| Enron spam, 2 | 98.7 | 97.7 | 98.0 | 99.6 (90%) |
| SST-2, 2 | 95.7 | 92.7 | 93.0 | 98.8 (82%) |
| AG News, 4 | 91.3 | 88.3 | 89.7 | 94.7 (88%) |
| Banking77, 77 | 76.0 | 78.7 | 81.7 | 89.9 (66%) |
Jev leads on three short-text sets and loses the 77-way one. Median time per call, 20 in flight: Jev 0.8 to 0.9 seconds; the two OpenAI models 1.4 to 5.0 seconds. Cost gaps in the post run from about 5 times (luna, few options) to 56 times (mini). Asking the LLMs for a probability per option is what inflates their bill; a choice-only call would be cheaper and would give you nothing to gate on.
The production half is the reason he wrote it. Three simple page gates on 1,005 pages each matched later outcomes 96 to 98% of the time. He put Jev in front of the existing classifier as a reject filter: under the line, skip the model. Replaying five runs with the filter on and off produced the same output on all 1,005 pages, with 50 to 67% fewer routing calls.
The hard page type did not work. “Does this page carry a value for one of 135 report rows” matched later outcomes 68.6%. Jev fired on mentions. The probability swung by 0.5 between runs on the same page. A general language model failed the same way, so he treats that as the task.
Inbox: 800 mails, 87.4% agreement with whether a human kept the task, 96% on the confident band (69% of the set). There is no lossless threshold. Drop everything under 0.03 and 78% of the non-requests disappear, at the cost of 5 of 231 real requests. Those five are mostly short forwards and mail whose only evidence is a photo.
His rule: trust the ends, hand the middle to the model you already run. Set the drop line from known positives, check it on data you did not use to set it. We did not rerun the public table or the pipeline.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 18 thread links the blog and the repository. The public table is 300 random items each from Enron spam, SST-2, AG News, and Banking77, same question and one-line label text for Jev, gpt-5.4-mini, and gpt-5.6-luna (OpenAI Responses API, reasoning low). Accuracy he reports: Enron 98.7 / 97.7 / 98.0; SST-2 95.7 / 92.7 / 93.0; AG News 91.3 / 88.3 / 89.7; Banking77 76.0 / 78.7 / 81.7. Jev median 0.8 to 0.9 s with 20 calls in flight. Pipeline numbers are author-run against later outcomes, not against another model: three page gates on 1,005 pages each (97.8%, 95.8%, 98.0% agreement), a 135-row extraction gate at 68.6%, and 800 inbox mails at 87.4%. Five production replays with the filter on and off matched on all 1,005 pages and cut routing calls 50 to 67%. Public cases and per-item answers are in jev-eval. Pipeline rows are not. We did not rerun either half. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; this story reports the write-up as posted.
Compare
Charly Poly also used Banking77 and beat open encoders on macro-F1 while losing on ECE. Kumar's 300-item slice is the same dataset against small generative models, and Jev loses that slice. Bryo's 1,565-email table is a different label set with a Gemini baseline. Vogel's 1,500-email clip has no accuracy. Kumar's contribution is the production replay with a lossless drop line on pages, a stated-loss line on mail, and a public scoring repo. The 16,000 figure is his count of API calls across both halves, not the size of one table.
Terms
- Reject filter
- Skip the slower model when Jev's P(yes) sits under a line set from known positives. On Kumar's page gates the line came from half the lowest P(yes) among real positives, checked by rotating production runs.
- Accurate when confident
- In this write-up, accuracy where a pick-one confidence is 0.9 or higher, or a yes/no probability is at or below 0.1 or at or above 0.9, plus the share of items in that band.
- Whole-document read
- A question whose answer needs the page as a unit, not a short snippet. Kumar's 135-row extraction gate is the example: Jev matched later outcomes 68.6% and the probability swung by 0.5 between runs on the same page.