jsort ranks a file by asking Jev which of two texts is more of something

Joe Weisenthal posted on September 22 that he had rescored the Fedlock corpus with Jev and JSort. The post says the corpus is 4,005 FOMC public speeches and statements since the mid-1990s, and it includes two charts. The tweet text gives the count and calls the result a glimpse of “intelligence too cheap to meter.” It does not print the chart series. A September 20 post from the same account had already scored Fed press conferences with Fedlock against JSort and named the developers as @eltokh7 and @TrippSmith_com.

The repo is keltokhy/jsort. Package metadata lists Khaled Eltokhy as the author, MIT, version 0.1.3. The installed command is jsort (uv tool install jev-sort). His September 21 reply is the link, under a note that the sort worked without task-specific fine-tuning.

jsort shows Jev two texts and asks one Noul: text A ranks higher than text B on a comparative you write, such as “more hawkish about inflation” or “easier to read.” A first round is a random ring. Later rounds pair texts that currently sit next to each other, so most pairs are never sent. The default -k 10 is about five questions per text. Answers are probabilities, so the fit is a fractional logit Bradley-Terry model. The printed score is in logits, centered on zero, with a robust standard error. A gap of 1 means Jev gives the higher text about 73% in a head-to-head. A gap of 3 means about 95%. Scores belong to that file and that wording unless you save an anchored scale and place later texts on it.

Two checks in the README were run on September 19 with Jev 1.13 through OpenRouter, uncached.

On CommonLit Ease of Readability, 300 random excerpts from a corpus whose teacher scores are themselves a Bradley-Terry fit, jsort -k 10 "easier to read" has Pearson r 0.824 and Spearman 0.841. That was 1,500 calls and $0.046. -k 16 lands on the same r at $0.074. A single “easy to read” probability per excerpt, the jgrep path, is r 0.754. SMOG, the best formula shipped with the corpus, is 0.661. The teacher scale’s reliability is about 0.78, and the README treats about 0.88 as the ceiling any measure can have against it. On pairs the teachers placed 0.25 to 0.5 apart, jsort agrees with them 67.5% of the time and the one-probability sort 63.2%.

On 95 FOMC opening statements, sorted whole on “more hawkish about inflation,” rank correlation with the funds-rate move announced that day is +0.46. Correlation with the change over the following 180 days is +0.37, on 91 statements. The six statements that announced 50- or 75-point hikes in 2022 are the top six. Mean score by chair is -2.19 for Bernanke (12 statements), -1.29 for Yellen (16), 0.62 for Powell (64), and 2.32 for Warsh (3). The README says the first live run was under half a minute and about seven cents, and that a cached replay asked zero new calls. This 95-file script is the measurement in the repo. It is a different count from the 4,005 speeches in Joe’s post.

A probe on every ordered pair of ten texts found that a two-option Choice and a Noul made the same decision 95.6% of the time. The Choice put 93% of its answers below 0.1 or above 0.9. The Noul left 28% of answers between those bands, which is the mass the scale is fit on. Short lines cost about seven cents per thousand at $0.042 per million input tokens. The default budget stops new questions at one dollar. Answers sit in ~/.cache/jev/answers.sqlite, the same file jgrep uses.

We did not install jsort or rescore either corpus.