On WANDS, more Jev questions move the mean to 0.610 and leave the median at 0.561

Nathan LeClaire posted on October 9, 2026, at 06:59 UTC, that Jev is a reranker, “and not just for text.” The post is that sentence. It has no chart, no corpus, and no question text. His bio names TypeSafe.

Yechan Do asked, three minutes later, whether multimodal input was coming. Opened the same day, the models page says Jev 1.13 takes text only: a string, a JSON object, or an array of text values. Images, audio, and video are not accepted. jev-latest and jev-preview both point at jev-1.13.0. On October 4 LeClaire had written that multimodal support would be good, and that he was not promising it.

The reply with a measurement is Ravi’s. He linked Doug Turnbull’s October 8 note, Bag of Decisions Reranker, and wrote that a better question than “is it relevant?” worked better. Turnbull has GPT-5 write yes/no questions for a single shopping query, sends them to Jev as Nouls, and adds the probabilities to a BM25 score.

The blog’s middle column is named jev_reranker. The CSV and the YAML call that row bag_of_decisions_direct. It is one configured question, “Does this candidate product satisfy the search query?”, with written criteria for true and false. The state includes the query, the title, and the description. The third column, bag_of_decisions on the blog and bag_of_decisions_example in the repo, is the GPT-5 list. Its state is the title and the description. The query sits in the generated questions. strategy.py refuses criteria on that LLM generator. Both configs name the model jev/jev-latest.

Dataset	Blog name	Repo strategy	Mean	Median
WANDS	bm25	bm25	0.540755	0.474620
WANDS	jev_reranker	bag_of_decisions_direct	0.574262	0.560945
WANDS	bag_of_decisions	bag_of_decisions_example	0.609814	0.560945
ESCI	bm25	bm25	0.289473	0.170709
ESCI	jev_reranker	bag_of_decisions_direct	0.313621	0.204850
ESCI	bag_of_decisions	bag_of_decisions_example	0.360079	0.341417

The two WANDS medians are one stored float, 0.5609447680982702. The means differ. The blog prints both medians as 0.561 and the bag mean as 0.610. Rounding the CSV mean 0.6098137673305334 to three decimals is 0.610. Rounding the BM25 mean 0.5407553553417934 to four decimals is 0.5408. The blog prints 0.5407.

The note at research/decisions/ecom.md says an LLM generated it. Its ESCI bag median is the string 0.34141707368146015. The CSV median is 0.34141715214740553. The strings share 0.341417 and then diverge. The note’s median and its mean, 0.36007937368146015, both end in the digits 7368146015. The blog’s 0.341 still matches the CSV at three decimals.

The published commands and scripts/run_ecom_decisions_variants.sh use 480 WANDS queries, 1,000 ESCI queries, and seed 42. The CSV stores the commit 8734fde3b23ce17eafdf60b2f55dfa6bced47e43 and does not store the query count. The note’s commands pass 8 workers. The script’s own default is 16. tool_calls_mean is 1.0 on the BM25 rows too, so that column is not a count of Jev calls. The CSV has no price.

The score is the retrieval score plus 10 times the sum of the Noul probabilities. decision_reranker.py adds a probability only when it is finite and between 0 and 1, so a missing answer drops out of the sum. All questions for one product go in one request. The strategy rescores the top 100 hits and writes those scores back. Every other product keeps its retrieval score. The metric then reads the top 10.

cheat_at_search keeps ranks through 10. The discount is 1/log2(2^rank), which equals 1/rank. Gain is (2^grade - 1) / rank. The ideal puts the dataset’s maximum grade in every slot from 1 to 10, and that ideal is the same for every query. A query with few relevant products cannot reach 1.

The two Jev rows share a first stage: title weight 9.4, description weight 4, k1 1.2, b 0.75. The printed BM25 row does not. configs/ecom_base/bm25.yml sets the title boost to 9.3 and the description boost to 4.1. The note calls 9.4 and 4 the baseline. On WANDS, the matched comparison is direct against the bag: mean 0.574262 to 0.609814, median unchanged. The gap against the printed BM25 row also changes the field weights.

The repo does not publish the per-query scores, so this desk could not see which queries moved. There is also no column that ranks by the summed probabilities alone. The blog says that sum could be used on its own. The six stored rows all add it to retrieval.

The other replies in the thread do not name a corpus. Oscar Romero wrote that their production recall was 5 to 10 points under a qwen3-reranker, and that Jev helps drop off-topic hits without much fine-tuning. Hev’s September table, a different test, prints mean nDCG@10 of 0.501 for a Jev batch reranker and 0.478 for Qwen3-Reranker-0.6B, across SciFact, NFCorpus, and FiQA. adaxon wrote that it was not able to beat a tuned light gbm, with no table. 3THER_ wrote that a guardrail can ride in the same request. TypeSafe’s classifying cookbook already asks an injection Noul beside relevance. That result stays on Jev RAG.

Qdrant posted the same day. On 100 NFCorpus queries, a Jev Score lift over BGE-small is +0.0665 at depth 30 and +0.0741 at depth 50. Their Iterative method uses 10 requests and reaches +0.0746 and +0.0789. On 240 WANDS queries their Jev rerank prints nDCG@10 0.769 against hybrid search at 0.683, and the number of product classes in the top 10 falls from 3.17 to 2.41. That 0.769 is a different pipeline from Turnbull’s 0.610.

This desk read the CSV, the configs, the scorer, and the NDCG function. It did not rerun the searches.