Published Oct 13, 2026

Updated Oct 14, 2026

On 40 client requests, Jev and Opus both scored 37 of 40, at $0.0015 against $0.005 to $0.015

This topic was created 13 days ago, and the information it contains may have evolved or changed since then.

Akshay Pachaar's TrueForge agent scored 37 of 40 client-issue matches with Jev and 37 of 40 without it. The printed decision costs are $0.0015 and $0.005 to $0.015. Review caught 5 wrong Jev matches. On October 13 he wrote that 40 is too small to establish how reliably confidence routes errors to review. On October 14, 2026, at 02:34 UTC, this desk's jev-1.13.0 call picked the intended issue at confidence 1.0. A trap shortlist scored 0.51. This desk did not install TrueForge.

Akshay Pachaar posted on October 13, 2026, at 07:30 UTC, that he put Jev inside TrueForge. The recap names a weekly client follow-up agent that matches client requests to GitHub issues. He tested 40 requests. The quoted write-up is the October 12 article, posted at 16:12 UTC, titled I put Jev inside an Open Agent Harness.

The October 12 article prints a table for those 40 requests. Both arms scored 37 of 40 on the right issue and the right status. With Jev, 16 answers went for a second look, and that review caught 5 wrong matches. Without Jev, 5 answers went for a second look, and review caught none. The printed cost of the decisions themselves is $0.0015 with Jev and $0.005 to $0.015 without it.

The October 13 recap calls that 150 times cheaper than letting the LLM make every match. The article estimates Opus at $4 per million input tokens and $20 per million output tokens would cost 150 to 250 times as much as Jev. With extended thinking on, it says the upper end is more likely. 150 times $0.0015 is $0.225. That estimate is not the without-Jev cell.

Jev’s list price in the article is $0.042 per million input tokens, output free. Each request is about 900 tokens, so 40 requests are about 36,000 tokens, well under one cent. 36,000 times $0.042 per million is $0.001512, which is the $0.0015 cell.

The company log has 40 requests. GitHub has 300 issues, 200 open and 100 closed, at about 230 tokens an issue, about 69,000 tokens. 40 requests against 300 issues is 12,000 pairs. The state and its longest question must fit in 32,000 tokens, with at most 255 options, so the backlog will not fit. A script keeps the 10 issues that share the most words. The article says that ranking misses some semantic matches. Jev then chooses among those ten, or none. The 40 requests went in one batched call.

The generation model drafts a new GitHub issue when Jev returns none, and reviews matches below the confidence cut. Open matches accept at 80%. Closed matches accept at 90%. After a pick, a script maps GitHub fields to a status the account team can act on: in progress if the issue is open with an assignee or an “in progress” label; tracked, not started if it is open with neither; fixed or declined if it is closed.

The test set covers straightforward matches, near-duplicates, regressions, feature requests, and requests with no matching issue. Each request has a known answer.

Both arms used Claude Opus 5.5 with default settings. The Jev agent is weekly-client-follow-up. The control is weekly-client-follow-up-no-jev. Screenshots of those configs name anthropic-claude-opus-5-5 at reasoning low, GitHub, Notion, and, on the Jev arm, jev-mcp. Jev is reached through Composio. The prompt names JEV_EVALUATE_STATE, run with COMPOSIO_MULTI_EXECUTE_TOOL, batched, up to 50. It says Jev cannot run inside a script. The harness holds writes for approval. Credentials stay outside the sandbox.

REQ-004 in the article is the failure the review is for. The customer says search keeps spinning and then gives up now that the whole company uses the product. GitHub has an issue about search timing out, with a different cause. Without Jev, the generation model picked that issue. With Jev, the top candidate scored 0.40 and went to review, which marked the request as new. REQ-001 is one of Jev’s three remaining misses. The wrong issue had the highest confidence, so the answer went through.

On October 13, at 09:17 UTC, Akshay replied that none was an explicit option. An uncertain none went to Opus for review. An accepted none led to a draft new issue. That handles missing options. One wrong Jev match still passed through without review.

At 09:19 UTC he replied that a lookup or rules would be simpler if requests shared stable keys or predictable wording. Clients and engineers describe the same problem differently. He would not recommend Jev or an LLM where a simple rule-based lookup is enough.

At 09:21 UTC he replied that the 40-request test is too small to establish how reliably confidence routes errors to review. Review caught 5 wrong matches, and one confident wrong match passed through. At 400, he would measure both errors caught and errors accepted at each threshold. No 400-request table was published.

The issues live in sitammeur/trueforge-jev-backlog. Opened October 14, 2026, the README says 300 synthetic issues, 200 open and 100 closed, generated to look like a mid-size SaaS backlog. They are not real bugs, and no real customer data is involved. GitHub’s issue search for that repo returned 306. The license field was empty. Commit 7d3bb073e3c3, October 1, 19:55 UTC, message Add README. The README says the generator, prompts, and analysis scripts are published separately with the article. This desk did not find that second repository.

TrueForge is TrueFoundry’s agent harness. Opened October 14, LICENSE is MIT, Copyright 2024-2026 TrueFoundry. npm @truefoundry/trueforge 0.178.0 requires Node 22.14 or newer. The article says Node 22.13 or newer. Local mode is npx @truefoundry/trueforge, one process on SQLite, at http://localhost:8790. Hosted mode uses Postgres and Redis. The README names Daytona as the sandbox. This desk did not run that command.

TrueForge’s benchmarking page prints DevRev’s Enterprise-Bench, 14 L1-L2 tasks, n = 3, same model and tools. Claude Managed Agents on Opus 4.8 solved 10.7 of 14 at $11.8 a run and 10.0M tokens. TrueForge on Opus 4.8 solved 10.7 of 14 at $8.6 and 3.7M tokens. deepagents on Opus 4.8 solved 10.0 of 14 at $21.2 and 16.5M tokens. 10.0M divided by 3.7M is 2.70. 3.7M divided by 16.5M is 0.224. The October 12 article’s 2.7 times fewer tokens, and under a quarter of deep agents’ tokens, match that table. The October 13 recap says roughly a third of the tokens. That bench is not the 40-request match.

Composio’s Jev toolkit, opened October 14, lists two tools: Evaluate State and List Models. The agent screenshot prints jev-mcp with 3 tools. Those two counts were not reconciled here.

This desk did not install TrueForge, Composio, or Daytona. On October 14, 2026, the Date header 02:34:24 GMT, a POST to https://api.typesafe.ai/v1/systemone with model jev-latest returned HTTP 200 and named jev-1.13.0. The state is a client request written here: global search in the web app spins for about a minute after the company crosses a few thousand seats, then the spinner stops with an empty result. The shortlist of ten issues plus none includes ISSUE-1406, an open unassigned ticket about that hang on a large tenant, and ISSUE-1182, an in-progress timeout on the public marketing site caused by an analytics script.

Choice was ISSUE-1406 at probability 1.0, confidence 1.0. Urgency noul was 0.85. Usage was 837 input tokens and 174 output tokens. The desk clock was 483 ms. There was no processing-time header. List-price arithmetic at $0.042 per million input tokens, output free, is $0.000035. The article’s 80% cut for an open issue would accept that answer.

A second POST, Date header 02:35:12 GMT, used the same request and dropped ISSUE-1406. The trap shortlist still had ISSUE-1182 and ISSUE-844, a closed API search 504 fixed by raising a gateway timeout. Choice was ISSUE-844 at 0.55, none at 0.43, confidence 0.51. Usage was 769 input tokens and 155 output tokens. The desk clock was 509 ms. List price $0.000032. ISSUE-844 is closed, so the article’s 90% cut would send it to review.

This site's reading

Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.

Verify

Primary post is Akshay Pachaar, @akshay_pachaar, October 13, 2026, 07:30 UTC, status 2109909532352917658. The text says he put Jev inside an open agent harness, 150 times cheaper decision-making than an LLM. Stack: TrueForge, Jev, Daytona. On 40 requests the agent was as accurate as letting Opus make every match. Low-confidence matches went back to Opus, and that review caught 5 wrong matches. TrueForge GitHub is linked. The post quotes the October 12 article and attaches a video of about 38 seconds. The still on this page is the 40-request table from that article.

The October 12 post, 16:12 UTC, status 2109678653974155459, prints the same 40-decision claim and links https://x.com/i/article/2109665778173935617. Opened October 14, the article title is I put Jev inside an Open Agent Harness. The article names weekly-client-follow-up and weekly-client-follow-up-no-jev, both on Claude Opus 5.5 with default settings. Shortlist of 10 issues per request. Accept at 80% confidence, or 90% for a closed issue. The printed table: both arms 37 of 40 correct on the right issue and the right status; second look 16 with Jev and 5 without; wrong answers caught 5 and 0; decision cost $0.0015 and $0.005 to $0.015. The article estimates Opus at $4 per million input tokens and $20 per million output would cost 150 to 250 times as much as Jev. 150 times $0.0015 is $0.225. That estimate is not the without-Jev cell. Jev list price in the article is $0.042 per million input tokens, output free, about 900 tokens a request, about 36,000 tokens for 40, well under one cent. 36,000 times $0.042 per million is $0.001512. The test set covers straightforward matches, near-duplicates, regressions, feature requests, and requests with no matching issue. Each request has a known answer. After a pick, a script maps GitHub fields to in progress, tracked, not started, or fixed or declined. The 40 requests went in one batched call. The state and its longest question must fit in 32,000 tokens, with at most 255 options.

Three replies from Akshay on October 13, under the recap. Status 2109936457985229199, 09:17 UTC: none was an explicit option; an uncertain none went to Opus; an accepted none led to a draft new issue; one wrong Jev match still passed through without review. Status 2109937173101412364, 09:19 UTC: a lookup or rules would be simpler if requests shared stable keys or predictable wording; clients and engineers describe the same problem differently; he would not recommend Jev or an LLM where a simple rule-based lookup is enough. Status 2109937444540018708, 09:21 UTC: the 40-request test is too small to establish how reliably confidence routes errors to review; at 400 he would measure errors caught and errors accepted at each threshold. No 400-request table was published.

REQ-004 in that article: the client says search keeps spinning and then gives up after the whole company uses the product. GitHub has a search-timeout issue with a different cause. Without Jev, Opus picked that issue. With Jev, the top candidate scored 0.40 and went to review, which marked the request as new. REQ-001 is one of Jev's three remaining misses: the wrong issue had the highest confidence and went through. Agent screenshots name sitammeur/trueforge-jev-backlog, anthropic-claude-opus-5-5 at reasoning low, GitHub, Notion, and jev-mcp. Jev is reached through Composio. The tool name on the prompt is JEV_EVALUATE_STATE, run with COMPOSIO_MULTI_EXECUTE_TOOL, batched, up to 50. The prompt says Jev cannot run inside a script. The no-Jev agent has GitHub and Notion only.

Opened October 14, 2026, github.com/truefoundry/trueforge LICENSE is MIT, Copyright 2024-2026 TrueFoundry. npm @truefoundry/trueforge 0.178.0, engines node >=22.14.0. The article says Node 22.13 or newer. Latest commit on main, c6ef7d19fb49, 05:43 UTC October 14, "bugfix: fix session environment handling in chat history (#1049)". Docs at trueforge.dev. Local run is npx @truefoundry/trueforge, SQLite, http://localhost:8790. Hosted mode uses Postgres and Redis. README names Daytona as the sandbox provider. This desk did not run that command.

TrueForge's benchmarking page, opened October 14, prints DevRev Enterprise-Bench L1-L2, 14 tasks, n = 3, same model and tools. Claude Managed Agents on Opus 4.8: 10.7 of 14, $11.8 a run, 10.0M tokens. TrueForge on Opus 4.8: 10.7 of 14, $8.6, 3.7M tokens. deepagents on Opus 4.8: 10.0 of 14, $21.2, 16.5M tokens. 10.0M divided by 3.7M is 2.70. 3.7M divided by 16.5M is 0.224. The October 12 article's 2.7 times fewer tokens and under a quarter of deep agents' tokens match that table. The October 13 recap says roughly a third of the tokens. GLM-5.2 on TrueForge is a different row, 11.7 of 14, $3.0, 3.8M tokens. This is not the 40-request match.

sitammeur/trueforge-jev-backlog README, opened October 14, says 300 synthetic issues, 200 open and 100 closed, for the article How to Run Jev Inside an Open Agent Harness. They are not real bugs. GitHub's issue search for that repo returned 306. The license field was empty. Commit 7d3bb073e3c3, October 1, 2026, 19:55 UTC, message Add README. The README says the generator, prompts, and analysis scripts are published separately. This desk did not find that second repository.

Composio's Jev toolkit page, opened October 14, lists two tools: Evaluate State and List Models. The agent screenshot prints jev-mcp with 3 tools. Those two counts were not reconciled here.

This desk did not install TrueForge, Composio, or Daytona. On October 14, 2026, Date header 02:34:24 GMT, a POST to https://api.typesafe.ai/v1/systemone, model jev-latest, returned HTTP 200 and named jev-1.13.0. The state is a client request written here about global search spinning for about a minute after a few thousand seats, then stopping empty. The shortlist of ten issues plus none includes ISSUE-1406, the same hang on a large tenant, open and unassigned, and ISSUE-1182, a marketing-site timeout caused by an analytics script. Choice was ISSUE-1406 at probability 1.0, confidence 1.0. Urgency noul 0.85. Usage 837 input tokens and 174 output tokens. Desk clock 483 ms. No processing-time header. List-price arithmetic $0.000035. A second POST, Date header 02:35:12 GMT, dropped ISSUE-1406. Choice was ISSUE-844, a closed API search 504, at 0.55, none at 0.43, confidence 0.51. Usage 769 input tokens and 155 output tokens. Desk clock 509 ms. List price $0.000032. ISSUE-844 is closed, so the article's 90% cut would send it to review. Key, request JSON, and request ids are not recorded here.

Compare

LangChain's langchain-typesafe harness is a Python Runnable plus experimental middleware. This page is TrueForge, an MIT TypeScript harness, with Jev reached through a Composio MCP tool on a 40-request match. The 37 of 40 table is Akshay's. It is not LangChain's 500/500 judge eval and not the cascade paper's 1.7 points.

The printed $0.0015 is the with-Jev cell on that table. The 150 times line is the article's Opus rate-card estimate. 150 times $0.0015 is $0.225. The without-Jev cell is $0.005 to $0.015. Do not merge those three figures. TrueForge's DevRev table, 10.7 of 14 on 3.7M tokens, is a third test. Akshay's 09:21 UTC reply says 40 is too small for the confidence cut. No 400-request table is on this page.

This desk's two calls used a request written here. They are not a rerun of the 40. 483 ms and 509 ms are this desk's clocks, not JevBench's p50. $0.000035 is list-price arithmetic on 837 input tokens, not a bill.

Terms

TrueForge: TrueFoundry's MIT-licensed agent harness. npm @truefoundry/trueforge 0.178.0, opened October 14, 2026, requires Node 22.14 or newer. Local mode is one process on SQLite.
weekly-client-follow-up: Akshay Pachaar's TrueForge agent that shortlists 10 GitHub issues per client request and asks Jev which issue matches, or none. Accept at 80% confidence, 90% if the issue is closed.
37 of 40: Correct matches, right issue and right status, on Akshay's 40-request test. Both the Jev arm and the Opus-only arm print this count. On October 13, 09:21 UTC, he wrote that 40 is too small to establish how reliably confidence routes errors to review.
$0.0015: Printed cost of the 40 Jev decisions on that table. The without-Jev cell is $0.005 to $0.015. The article's 150-to-250 times estimate is a separate Opus rate-card line.
0.51: Choice confidence on this desk's October 14, 2026 trap shortlist, Date header 02:35:12 GMT, jev-1.13.0. The pick was a closed API 504 issue at probability 0.55, with 0.43 on none.

On 40 client requests, Jev and Opus both scored 37 of 40, at $0.0015 against $0.005 to $0.015

This site's reading

Verify

Compare

Terms

Sources

Same tags