One day of grant posts, 100 of them, hid 75 of 78 rejects after the questions changed

Deji, @DejisHQ, posted a thread on September 28 about a grants bot he is rebuilding for early-stage founders. The old bot checked eight funding sources a day and let Llama 3.3 70B score each post out of 10. For the sort he is testing now, he took 100 posts from one day, labeled them himself with Jev’s answers hidden, and then marked Jev like a test. There is no repository and no item file. We did not see the 100 posts.

His key, written before the comparison: 7 of the 100 were a real yes for those founders. The rest were jobs, PhD places, news, and old crypto ads. The questions he lists are closed: can someone from the founder’s country apply, is it only for researchers, is it only for charities, is there a fee. Answers are yes, no, or the page doesn’t say. Rules on top of those answers hide a closed program, a country bar, and a researchers-only call. An unsure answer goes to a person.

He says the first try was close and wrong, and he names three causes. The model read short previews, so he switched it to the whole page. It mixed up “no” and “the page doesn’t say,” so that became its own answer. Batches of 40 confused it, so he sends one post at a time. He also says one of his own nos was the mistake: a food-tech listing he had skimmed, which he then kept as a yes after reading Jev’s answer. The thread does not print how many labels moved.

The counts he prints for the run after those changes: 75 of 78 posts he had called no were hidden, 5 of 5 real ones were shown, the same program listed on two sites was caught 6 of 6, and deadlines were found on 33 of 41. The day of testing cost about 2 cents. The thread does not print tokens, a model id past the name Jev, or a table of the other errors. The 7 real yeses and the later 5 of 5 are both his figures. They are not the same denominator.