Published
Updated
jev-code's three-task ladder still fails after the call-nesting fixes
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
Roberto Montalti posted rhighs/jev-code on September 26. Jev picks grammar productions. The program builds the AST and runs a compiler. The guide's ladder, one run per task, dated September 18, still fails guessing-game, file-io-script, and multi-file-package on commit 517d0f0. A hello-world smoke test does pass, in 3 turns and 16 requests. package.json is 0.2.0 and has no license field. The root listing has no LICENSE file. We did not run it.
rhighs/jev-code is a TypeScript SDK and CLI from Roberto Montalti, a software engineer at Kong who also studies at the University of Milan and the Polytechnic University of Milan. He posted it on September 26. The post says the project is fully open source. package.json is version 0.2.0 and has no license field. The root listing we opened has no LICENSE file. The package depends on @typesafe-ai/sdk at ^0.6.0 and wants Node 22 or newer. The post attaches a gif, so this page has no still. We did not run the eval.
Jev does not write the source file. It picks the next production in a grammar. Local code assembles an AST from those picks and hands the tree to a compiler. The languages in the guide are Python, Bash, JavaScript and TypeScript, C, Rust, Go, Lua, and Ruby. A run stops at 50 turns, 512 requests, or 5 minutes, whichever comes first. Python nesting stops at 8. A block stops at 16 statements. Bash commands the compiler emits are not sandboxed.
decide is the small CLI in front of one question. Its exit code is the index of the chosen option. A boolean returns 0 or 1 according to the threshold. A failed call returns 125.
The guide keeps a ladder of three tasks: a guessing game, a script that reads and writes a file, and a package split across files. The table is dated September 18, one run per task, and the guide says this eval does not run in CI. The command it shows passes --yes. On commit 517d0f0, after fixes for nested calls and for Bash, all three still fail. Guessing-game stops at 38 turns, 2,000 requests, and 10 minutes 44 seconds. The file script stops at 34 turns, 3,000 requests, and 13 minutes 24 seconds. The multi-file package stops at 60 turns, 2,516 requests, and 13 minutes 53 seconds. A search width of 3 fails the first task at 37 turns, 2,000 requests, and 11 minutes 4 seconds, fails the file script at 9 turns, 3,000 requests, and 7 minutes 49 seconds, and does not run the package.
The misses the guide quotes are picks that parse. randint(range(1 + 100, 100), 100) is not a draw from 1 to 100. print('Too low', None) prints a second value the task did not ask for. A greeting function is called with 'Name' where the task wanted 'World'. Earlier rows crashed. This row compiles the wrong program.
Two smoke tests on the same page do finish. Hello world takes 3 turns and 16 requests. The interactive repeat of that smoke records 20,426 input tokens. for i in range(5): print(i) takes 3 turns, 22 requests, and 9.6 seconds. A separate number-guessing trial spends its 256 requests and writes no file.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
rhighs/jev-code, Roberto Montalti. The September 26 post calls the project fully open source. package.json is version 0.2.0 and has no license field. The root listing we opened has no LICENSE file.
The SDK depends on @typesafe-ai/sdk at ^0.6.0 and asks for Node 22 or newer. Jev chooses grammar productions. Local code builds an AST and invokes the language compiler. Languages named in the guide: Python, Bash, JavaScript and TypeScript, C, Rust, Go, Lua, and Ruby.
Defaults: 50 turns, 512 requests, 5 minutes. Python nesting stops at 8. A block stops at 16 statements. Bash is not sandboxed. The decide command's exit code is the choice index, or 0 and 1 for a boolean at the threshold, or 125 when the call fails.
The guide's eval table is dated 2026-09-18, one run per task, and the guide says the eval does not run in CI. It also says the run implies --yes.
Latest row, call-nesting and bash fixes, commit 517d0f0: guessing-game fails at 38 turns, 2,000 requests, 10 minutes 44 seconds; file-io-script fails at 34 turns, 3,000 requests, 13 minutes 24 seconds; multi-file-package fails at 60 turns, 2,516 requests, 13 minutes 53 seconds.
Search width 3 also fails: guessing-game 37 turns, 2,000 requests, 11 minutes 4 seconds; file-io-script 9 turns, 3,000 requests, 7 minutes 49 seconds; multi-file-package was not run.
The guide's examples of the remaining misses are randint(range(1 + 100, 100), 100), print('Too low', None), and a greeting called with 'Name' where the task wanted 'World'. Smoke tests that pass are narrower. Hello world is 3 turns and 16 requests, and the interactive repeat records 20,426 input tokens.
A for-loop, for i in range(5): print(i), is 3 turns, 22 requests, and 9.6 seconds. A number-guessing trial exhausts 256 requests and writes no file. The post attaches a gif, so this page has no still.
We did not run the ladder.
Compare
mizchi's fit page counts 0 illegal moves in 489 enumerated MOBA decisions, where an illegal option cannot be written down. jev-code's host also enumerates productions, and the ladder still accepts productions that compile and do the wrong thing.
CLM-8B's DeepSWE holdout is 31 of 38 on a different coding task, with a verifier over sampled patches. The three failures here are one run each, on the author's machine, dated September 18.
Terms
- 517d0f0
- The commit on the latest ladder row, after call-nesting and bash fixes. Guessing-game, file-io-script, and multi-file-package all still fail. Search width 3 fails the first two and does not run the third.
- 125
- The decide command's exit code when the Jev call fails. A choice returns the option index. A boolean returns 0 or 1 at the threshold.
- 16 requests
- The hello-world smoke test, 3 turns. The interactive repeat of that smoke records 20,426 input tokens. The three-task ladder is a separate table, and none of those tasks pass.