Published
TypeSafe lists nine ways Jev 1.13 is still jagged, and asks for more
Cofounder Sasha Sheng posted that Jev is still too slow, too expensive, and too dumb, and pointed at TypeSafe's jaggedness page for jev-1.13. The docs, last reviewed September 17, name nine failure modes and ask people to file more in Discord.
Sasha Sheng, a TypeSafe cofounder, posted on September 18 that Jev is still too slow, too expensive, and too dumb. TypeSafe quoted her: “Let us know how Jev is bad!” The link is the Jev 1.13 jaggedness page, last reviewed September 17. A later post from Sheng asked people to file issues in the model-jaggedness channel on Discord.
The page applies to jev-1.13. It says the model is fast, calibrated, and good at common-sense judgment, and that it struggles with extra hops, literal wording, and numeric precision. Nine failure modes follow, each with a “do this instead.”
The short list: write the exact condition; keep arithmetic in code; extract dates, then compare in code; cut extra hops; filter state before the call; test adversarial text; align instructions with criteria; do not assume two phrasings of the same question agree; do not ask Jev to generate text.
Two tables on that page show the invariant problem. On the ticket “I’m not happy with the fit. What are my options here?”, a Noul for “Is the customer asking for a refund?” returns 0.22, while a yes/no Choice on the same question returns 0.01 yes and 0.99 no. On “I was charged twice for the same order. Can someone look into this?”, a Noul for a refund request returns 0.72 and a Noul for “something other than a refund” returns 0.47. Those two numbers sum to 1.19. The docs say not to carry a threshold tuned on a Noul over to a Choice, and not to hold the model to arithmetic identities between separate questions.
Counting is called out the same way. jev-1.13 does not tally characters, term occurrences, or items in a long list. The worked example iterates in code and asks one yes/no per item. Date comparison is the same split: extraction can be a Choice over months and days; ordering belongs in code.
The page also says accuracy falls as unrelated material piles into state, that adversarial content can steer the answer, and that chaining choices to force generation is slow and weak. For extraction, it recommends getting candidate strings from a regex or a generative model, then letting Jev pick.
The reminder at the bottom is to avoid questions code can compute, several judgments packed into one question, extra layers of indirection, and more context than the question needs.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Sheng's September 18 posts and TypeSafe's quote match the public docs at docs.typesafe.ai/model-jaggedness/jev-1.13, last reviewed September 17, applying to jev-1.13. The page lists nine failure modes. The numeric examples under structural invariants are from that page: on one ticket, a Noul for "asking for a refund" returns 0.22 while a yes/no Choice on the same question returns 0.01 / 0.99; on another ticket, a Noul and its negation sum to 1.19. Discord is named as the place to file new issues (model-jaggedness channel). We did not reproduce the examples.
Compare
Launch materials stress speed, price, and "can't hallucinate." This page is the company's list of the jobs jev-1.13 is bad at: counting, date arithmetic, extra hops, padded state, adversarial text, and asking it to write. Malis's Postgres overlay hit the world-knowledge versus catalog-label problem, which sits next to "keep arithmetic in code." Bryo's missing attachments sit next to "text only," already on the models page. The invariant table is the first official warning that a Noul threshold should not be reused on a Choice.
Terms
- Jaggedness
- TypeSafe's word for uneven failure modes on jev-1.13. The docs page lists nine of them and says many will be fixed in later versions.
- Literal reading
- Answering the words in the instruction rather than the intent behind them. The docs say to put boundary cases in the criteria, or split the judgment into two questions.
- Structural invariant
- A relationship you might expect between two questions, such as P(yes) on a Choice matching a Noul, or a Noul and its negation summing to 1. The docs say jev-1.13 does not guarantee those identities.