Published
AstroHan ran 30 coding tasks with a Jev context filter: 25 passed, 22 without it
AstroHan scored every tool result with Jev before it reached deepseek-flash, on the same 30 FrontierHarness tasks. Filtered 25/30 pass@1 versus 22/30 unfiltered. On meriyah the unfiltered run timed out at 0/49 tests; the filtered run passed 49/49. One run per arm. Token spend did not fall.
AstroHan posted on September 20 that people were pruning agent context with Jev and had not checked whether the agent still finished the task. The thread quotes Tamara Tran’s later compaction clip and then a 30-task run. The earlier compaction write-up on this desk is Tran’s keep-or-drop sieve.
The 30 tasks are the FrontierHarness Eval set. Same agent, same model (deepseek-flash), only the filter changes. One run per arm. pass@1: 25/30 with Jev on, 22/30 with it off. Cost per solved task $0.113 versus $0.127. The author says the filter lost two of those 30, and that the number is a direction, not a magnitude.
A DeepSWE chart of nine named tasks is titled “filtered context is never worse,” 6 wins, 0 losses, 3 ties, sign test p≈0.031. meriyah is the extreme: unfiltered, 1.46 million characters of tool output, the run drowned and timed out, 0 of 49 tests. Filtered: 49 of 49, at 60% of that cost. anko 9/9 versus 5/9. arktype 24/25 versus 23/25. katex 94/94 versus 92/94. python-statemachine 72/72 versus 70/72. httpx 121/122 versus 120/122. expr, fastapi, and scc tied at 79/79, 137/137, and 31/31. terminal-bench tied 19 of 21 both ways. Those tasks print about 8,000 characters, so there is little to drop. The gap shows up where output is huge.
The filter sits between the tool and the model. bash, read, and apply_patch return raw output. Jev gets one boolean per roughly 2,000-character chunk, with state of task, intent, action, and output. Keep if p > 0.5. Kept chunks go back into context verbatim. Dropped runs become one marker pointing at .jev-store/id.txt. A later read can pull the full text; that path is not filtered. The author’s diagram says nothing is destroyed.
A later post in the thread revises the cost story. Input tokens were 126 million with the filter and 123 million without, “same total cost.” The author wrote that Jev did not cut the token bill, and that the gain on this run was the agent not losing the thread on long tasks. VMs were sponsored by @runta. There is no repository in the thread. We did not rerun it.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 20 thread quotes Tran and posts four charts. The 30 tasks are the FrontierHarness Eval set. Same agent, same model (deepseek-flash), one run per arm. pass@1 25/30 with the filter, 22/30 without. Cost per solved task $0.113 versus $0.127. meriyah: unfiltered 1.46 million characters of tool output, timed out, 0 of 49 tests; filtered 49 of 49 at 60% of that cost. A DeepSWE chart of nine named tasks reads 6 wins, 0 losses, 3 ties, sign test p≈0.031: meriyah 49/49 vs 0/49, anko 9/9 vs 5/9, arktype 24/25 vs 23/25, katex 94/94 vs 92/94, python-statemachine 72/72 vs 70/72, httpx 121/122 vs 120/122, expr 79/79, fastapi 137/137, scc 31/31. terminal-bench 19 of 21 tied both ways; those tasks print about 8k characters. The author says the filter lost two of the 30. Input tokens 126 million versus 123 million, "same total cost." Filter: one boolean per ~2k-character chunk, keep if p > 0.5; dropped text stored and readable via `read`. VMs from @runta. No repo. We did not rerun it. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services; this story reports the posts as published.
Compare
Tran's September 17 compaction post, and fast-jev-compaction, argued for scoring tool calls instead of summarizing them. This is the first head-to-head on this desk that keeps the agent and the model fixed and only changes the filter, then reports task pass rates. Kumar's reject filter and Poly's 90% precision gate drop traffic before an expensive call; this drop happens after the tool has already run. LangChain's judge eval is a different job: frozen traces, not live coding. One run per arm, so the 25 versus 22 is a direction, as the author wrote.
Terms
- Context filter
- AstroHan's loop: Jev answers one boolean per ~2k-character tool chunk (state is task, intent, action, output) and the harness keeps the chunk when p > 0.5.
- pass@1
- Whether the task passed on the single attempt that was run. 25/30 with the filter, 22/30 without, on the FrontierHarness 30.
- meriyah
- A DeepSWE task whose unfiltered tool output was 1.46 million characters. Unfiltered 0/49 tests; filtered 49/49.