On eight Stripe passages, a 0-9 Score put Roll secret at 8.86 and the other six at 2.42 or below

Nathan LeClaire posted on October 12, 2026, at 02:41 UTC, a ranking tip for Jev. He wrote that a Score such as “Is this content relevant to this query?” is tempting, and that a Choice, “Which is the most relevant?”, over pairs, then something like Bradley-Terry, seems better. The attached graphic is two panels. The Score panel crosses out “How relevant is this? 0-10” and piles eight balls between 7 and 9. The Choice panel spreads A through H on a Bradley-Terry axis, marks H as number one, and shows 11 of 28 duels. His bio names TypeSafe.

deekay wrote that he was already scoring, because it is easier to picture, and that he would rethink relevance sorting. LeClaire replied, “try it, don’t sleep on Bradley-Terry. i’m getting good vibes.” Dan Raviv asked for a built-in. LeClaire asked @CompleteSkeptic whether he could build it. Tobias Petry asked for a TypeSafe ranking primitive: n values in ranked order, 1.0 for the winner, and a score in [0, 1] for how likely each other value had been to win. LeClaire tagged Ted Kalaw with “free product idea.” Opened the same day, the primitives pages still list Choice, Score, and Noul. The re-ranking cookbook still sorts a shortlist by one Noul per candidate.

Nick posted that he could add the method to question-kit. At 07:39 UTC he linked commit 25a3b71 and wrote that he would publish tomorrow. The package is MIT, version 0.2.0, author Nick Gideo. That commit lists rankByScore and rankPairwise under Unreleased. rankPairwise sends one two-option Choice per pair in a single request, alternates which item is offered first, and fits Bradley-Terry with Hunter’s MM algorithm and prior 0.1. The method comment says that in their evals, sorting a Score by its expected level ordered lists about as well as rankPairwise, and better on some, for fewer questions.

Didier Lopes wrote that pairwise Jev was one of the best uses he had found, and that he had built an Elo leaderboard of tickers from multi-agent fundamental pitches. The quote has no table. Vladimir Kalnitsky wrote that pairwise comparison, at least in testing, shows whether the ordering is associative and transitive. Darin wrote that a dense cluster is a sign of rubric weakness. John Kovacs wrote that sorting on a relevance score still feels cleaner until two near ties flip under Bradley-Terry.

jsort already ranks a file from pairwise Noul judgments and a Bradley-Terry scale. TypeSafe’s Score page says a Score is a position on levels you describe, and that levels that are only numbers leave the model with nothing to match. The API accepts at most 10 levels. This desk used “0” through “9” for the numeric Score, so that scale stands in for the graphic’s 0-10.

This desk wrote two lists of eight passages, labeled A through H, against the query “How do I rotate a Stripe webhook signing secret?” H is the Roll secret steps. G is the signing-secret page and the note that rolling invalidates the old secret. The intended order is H, G, F, E, D, C, B, A. That order is this desk’s. The texts are not a retrieval corpus. Calls went to POST https://api.typesafe.ai/v1/systemone with model jev-latest. Every response named jev-1.13.0 and returned HTTP 200. Each list used two requests: 24 per-item questions, then 28 pairwise Choices. First-offered option and the Bradley-Terry fit followed question-kit.

The Stripe shortlist keeps every passage on Stripe keys or webhooks. A is the trap: it rotates the API secret key.

Item	Numeric 0-9	Descriptive 0-3	Noul	Pairwise strength	Soft wins of 7
H Roll secret	8.86	2.98	0.96	2.46	6.92
G Signing-secret page	8.07	2.38	0.51	1.69	6.08
F HMAC verification	2.42	1.01	0.02	-0.05	3.46
A API secret key	1.74	0.93	0.03	0.30	4.05
E Create endpoint	1.74	0.97	0.02	0.10	3.71
B stripe listen	1.58	1.00	0.02	-0.60	2.54
D HTTPS and retries	0.62	0.91	0.01	-1.66	0.96
C Restrict events	0.50	0.79	0.01	-2.23	0.28

The 0-9 Score on that list ran from 8.86 to 0.50. The two rotation items sat at 8.86 and 8.07. The other six sat at 2.42 or below. Spearman against the intended order is 0.690. The graphic’s pile between 7 and 9 did not show up on these eight texts.

The four-level Score put H at 2.98 and G at 2.38. The other six sat between 0.79 and 1.01, on the “related, but does not answer” rung. Spearman 0.738. That middle band is the cluster Darin’s reply is about.

The Noul left H at 0.96 and G at 0.51. The other six sat at 0.03 or below. Spearman 0.357. That is the cookbook shape: one yes/no per candidate, then sort.

Pairwise Choice ordered H, G, A, E, F, B, D, C. Spearman 0.548. H beat G at probability 0.92. A, the API-key trap, ranked third, with 4.05 soft wins of 7. The majority graph had a cycle through A, F, and E: A over F 0.89, F over E 0.58, E over A 0.70. Score cannot report that cycle. Desk clocks on the two requests were 455 ms and 419 ms.

A second list mixed in Lisbon weather and a PostgreSQL password reset. Numeric and descriptive Score both ordered H, G, F, E, D, A, C, B, Spearman 0.929. The 0-9 scores were 8.87, 8.01, 3.81, 1.81, 0.45, 0.43, 0.06, 0.02. The Noul again left six items at 0.01 or 0.02. Pairwise ordered H, G, F, E, A, D, C, B, Spearman 0.857, with no cycle. “Click here for more information” beat the Stripe refund paragraph at 0.65.

Four calls used 9,664 input tokens and 2,374 output tokens. At $0.042 per million input tokens, output free, that is $0.000406. The pairwise request asked 28 questions. The per-item request asked eight of each type. Migtronix’s reply in the thread was that this is how you get charged for input tokens.

On these two lists the descriptive Score Spearman was higher than pairwise Bradley-Terry. question-kit’s unreleased note already prints that ranking for its own evals. Pairwise Choice ranked the API-key trap third and returned a cycle. TypeSafe has not shipped a ranking primitive.