Published
SREGym put Jev on a Codex agent: 24 passes out of 50, up from 20
Jackson Clark's team wired Jev into SREGym-Lite as a test ranker and evidence reviewer on a Codex agent that still diagnosed and repaired. With gpt-5.6-luna, the Jev-assisted agent passed 24 of 50 attempts against 20 without it. Two problems got worse.
Jackson Clark posted results from wiring Jev into SREGym, a live incident benchmark from researchers at the University of Illinois and collaborators. The work is written up on the SREGym blog. Coauthors named on the thread include Saad Mohammad Rafid Pial, Yiming Su, and Tianyin Xu.
Jev did not diagnose the incidents. The agent still inspected the cluster and applied repairs. Two tools sat in the Codex loop:
jev_plantook three to five competing hypotheses plus a read-only test for each, added a bounded snapshot of the namespace, and asked Jev to rank the tests.jev_submitreviewed evidence before a diagnosis or a mitigation went to the grader. A diagnosis needed a causal mechanism and a current failure. A mitigation needed evidence that the repair addressed the cause, restored function, and looked durable.
Every required review question had to reach probability 0.70. A rejected submission sent the agent back to jev_plan instead of letting it reword the same claim.
They ran gpt-5.6-luna with and without those tools on 10 SREGym-Lite problems, five attempts each. Without Jev the agent passed 20 of 50 attempts. With Jev it passed 24 of 50. Four problems improved, two got worse, four stayed the same. The authors say five attempts per problem is not enough to treat the eight-point gap as a general gain.
The largest gain was internal_traffic_policy_local_astronomy_shop, from 0/5 to 3/5. The Service had internalTrafficPolicy: Local while its only endpoint sat on another node. Baseline runs chased OpenTelemetry errors. After jev_submit rejected a wrong causal story about the frontend listener, later tests tied timeouts to the Service policy. edge_request_filter_cpu_saturation moved from 2/5 to 4/5. Network-policy and DNS-policy each gained one extra pass.
Where it failed, Jev often accepted a cluster that looked healthy now. On a namespace memory limit, agents added requests to the rejected workload and left the quota in place. On a rolling update, they restored three Ready replicas and left maxUnavailable: 100% with maxSurge: 0. On other problems the agent never proposed the right hypothesis, so ranking the remaining tests could not recover it.
They did not test repeated votes, asking Jev after every tool call, or a check on whether a mitigation is safe to apply. They measured pass rate. They did not measure time to diagnosis.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The thread and the SREGym blog agree on the setup: Codex harness, gpt-5.6-luna, 10 SREGym-Lite problems, five attempts each, 20/50 without Jev and 24/50 with Jev. Tools are named jev_plan and jev_submit. Required review questions used a 0.70 probability threshold. We did not rerun the benchmark. Pass rate is the published metric; the authors say they did not measure time to diagnosis. Five attempts per problem is the sample they used, and they describe the eight-point lift as encouraging rather than a general result.
Compare
Browser Use and Stagehand also keep Jev off the keyboard: the host proposes legal actions, Jev ranks them, the agent still clicks. Here the legal actions are diagnostic tests and the click is a cluster investigation. LangChain's AutoModeMiddleware refuses a tool call; jev_submit refuses a diagnosis or mitigation until evidence clears the threshold, then sends the agent back to plan. Every's writing tests score documents in a batch. This test scores whether an agent finishes an incident.
Terms
- SREGym-Lite
- A smaller SREGym problem set meant to run on modest clusters. This experiment used 10 of those problems, not the full 21-problem lite suite.
- jev_plan
- A tool the agent calls with three to five hypotheses and a read-only test for each. Jev ranks those tests against a fresh namespace snapshot and does not run them.
- jev_submit
- A review gate before diagnosis or mitigation is sent to the grader. Every required question had to reach probability 0.70 or the agent had to gather new evidence.
- Pass rate
- Successful attempts out of five per problem. The authors used this instead of time-to-diagnosis.