Glean replayed four internal decisions on Jev, and 40 expert transfers were 8.1 times faster

Tony Gentilcore posted an article on September 28 about four bounded decisions Glean already runs. The byline lists @eddiedzhou, @mr_cheu, @MatZhao, Aviral Singh, and Manav Agrawal. The article says the team expects a large cost and latency gap wherever the baseline is a language model, and that the quality result depended on the task. Several of the charts did not come through as text when we copied the article. The figures below are the ones in the prose.

Query classification maps a request onto a task label the rest of the system already uses. Glean scored agreement with its production language-model labels, and also ran Laya and a Laya checkpoint fine-tuned locally in a few hours. The prose says Jev beat the untuned Laya and that the fine-tune beat Jev. It does not print the agreement rates in the paragraphs we could copy. The trade the article names is labels and a training setup, against calling a hosted model when the taxonomy is still moving.

Model routing, in the version they replayed, picks among three experts. They ran 751 comparable golden entries through a Jev router and compared the route to the existing prompt router and to the golden label. The accuracy percent is in a chart we do not have as text. They also took the 40 entries where the old path actually transferred to an expert and timed those calls. The median speedup was 8.1 times. The article notes a structural difference: if Jev does not transfer, the system has spent an extra call, while the production router can start the work inside the same first language-model call. The 8.1 times is an offline sample of 40 transfers.

Reranking asked Jev to reorder up to 50 search results. They tried four ways of asking. Choice was the strongest. On 4,855 captured queries, with production order removed as a tie-break, Jev Choice cost about $0.00044 per query and took 0.195 seconds at the median. It gave the same score to about 37 of the 41 candidates in an average query. Putting production order back, only to break those ties, raised Recall@6 from 40.2% to 44.3%. The article says the uncorrected number looks stronger than Jev’s own scores, and that the evalset user cannot see every canonical document, so the absolute recall is not the production ranker. The direction they keep is that Jev is a cheap baseline on this set.

Citation checking used one production-eval answer, 1,448 words, held fixed with its evidence. Jev and GPT-5.6 Luna, at no reasoning and at xhigh, each judged it three times. The precision and recall percents are in a chart we do not have as text. What the prose does print is consistency on 28 paragraphs. A change means the judge flipped its verdict on whether a paragraph had enough citation coverage in at least one of the three runs. Pairwise, that is 84 comparisons per judge. Jev’s only flip was whether one paragraph needed a citation at all. Its recall category did not change. The article says xhigh roughly tripled Luna’s model time and the Luna runs still flipped more often. It also says this is not an accuracy result: the judges used different support rules, and nobody labeled the paragraphs by hand. Jev’s clock in that test is sequential client time. Luna’s is the sum of model-call durations. The costs include caching.

The article says Glean plans to ship Jev on some of these paths after operational work, including data residency, and that an internal hackathon the same week would try more of them. We did not rerun the four sets.