Published
Updated
A chart puts Jev's reading of 25 probability words next to 99 people
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Mike Taylor posted a chart of 25 phrases from a 2026 survey of 99 people. The caption says Jev 1.13.0 answered one Choice per phrase, from 0 to 100 in steps of 5, and that rank agreement with the human order is 0.99. The median gap is 3 points. We read the image. The follow-up links a Claude artifact. The Hails essay, opened September 30, is that survey of 99 people. Probable is 70 in its median table. The essay has no Jev column.
Mike Taylor posted a chart on September 22 of how Jev reads 25 probability phrases. The image title is “Does Jev read probably the way people do?” The caption says the ridges are 99 people’s answers from Hails, 2026, and the diamonds are TypeSafe’s Jev on the same phrases, with each probability split into 99 marks so it can sit on the same drawing. Rank agreement is 0.99. The median gap is 3 points, which the caption says is about the same as the gap between two human surveys.
The footer names the sources. Human data: D. Hails, “The CIA was ‘Probably’ Right” (2026), n=99. Model: TypeSafe jev-1.13.0, one Choice question per phrase, answers from 0 to 100 in steps of 5. Both series are smoothed with a Gaussian kernel. The phrases run from Certain, Almost Certainly, and Highly Likely down through Probably, About Even, and Unlikely to Almost Certainly Not and Impossible. The follow-up post links a Claude artifact. Sasha Sheng replied that the chart is very pretty. That reply does not add a method note.
TypeSafe’s calibration claim is that an 80% answer should be right about 80% of the time. This chart compares word meanings: where people put “probable,” and where Jev puts it. The caption does not print a mean for each phrase. The 0.99 and the 3-point median are the numbers written on the image. Taylor’s earlier 777 writing judgments score prose, not these hedges. We read the image. We did not ask Jev the 25 questions.
On September 30, 2026 we opened The CIA was “Probably” Right. The essay recruited 99 English-speaking respondents with an undergraduate education or higher, through Prolific. Each person saw 25 phrases in a random order and moved a slider.
The phrases are Barclay’s 16, plus nine more, including frequent, realistic possibility, rare, and occasional. Three control items were left out of the analysis. The printed medians are rounded to the nearest 5.
Almost Certainly is 85, 90, and 90 in the columns for NATO officers in the 1970s, Zonination 2015 (n=46), and Hails 2026. Probable is 70, 70, and 70. Likely is 75, 70, and 75. The essay says probable has meant about 70% for half a century. The page is the human survey. It has no Jev column.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Primary post is @hammer_mt, September 22, 2026, 22:46 UTC. The image title is "Does Jev read probably the way people do?" Caption: ridges are 99 people's answers for 25 probability phrases (Hails, 2026).
Diamonds are TypeSafe's Jev on the same phrases, its probability split into 99 to match. Rank agreement 0.99. Median gap 3 points, about the same as between two human surveys. Footer: human data is D. Hails, "The CIA was 'Probably' Right" (2026), n=99.
Model is TypeSafe jev-1.13.0, one Choice question per phrase, 0 to 100 in 5% steps. Both series are smoothed with a Gaussian kernel. Phrase labels on the chart, top to bottom: Certain, Almost Certainly, Highly Likely, Very Good Chance, Likely, Probable, Frequent, Probably, We Believe, Realistic Possibility, Better Than Even, About Even, Occasional, We Doubt, Probably Not, Unlikely, Chances Are Slight, Improbable, Little Chance, Highly Unlikely, Remote Chance, Rare, Almost No Chance, Almost Certainly Not, Impossible.
The follow-up links a Claude artifact. Sasha Sheng replied "It's very pretty!" We read the posted image. We did not rerun the 25 questions.
On September 30, 2026 we opened the essay the footer cites. It recruited 99 English-speaking respondents with an undergraduate education or higher, through Prolific. Each person saw 25 phrases in a random order and answered on a slider.
The set is Barclay's 16 phrases plus nine more, including frequent, realistic possibility, rare, and occasional. Three control items were left out of the analysis. Printed medians are rounded to the nearest 5.
Almost Certainly is 85, 90, and 90 across NATO officers in the 1970s, Zonination 2015 (n=46), and Hails 2026. Probable is 70, 70, and 70. Likely is 75, 70, and 75. The essay says probable has meant about 70% for half a century. The page is the human survey. It has no Jev column.
Compare
TypeSafe's calibration claim is that a stated probability should match the rate of correct answers. This chart is a different comparison: where Jev places an English hedge, against where 99 people placed it. The 777 writing judgments score labels on prose.
This chart scores the words. No per-phrase mean is printed in the caption, so the 0.99 and the 3-point gap are the figures the image states.
Terms
- rank agreement 0.99
- The chart caption's summary of how Jev ordered 25 probability phrases against the human survey. The median gap printed beside it is 3 points.
- Hails 2026
- D. Hails, The CIA was 'Probably' Right, opened September 30, 2026. A Prolific survey of 99 English-speaking respondents with an undergraduate education or higher, 25 phrases, medians rounded to the nearest 5. Probable is 70 in the Hails column, and 70 in the NATO 1970s column and the Zonination 2015 column (n=46). The essay has no Jev column.