Published
Updated
A CUA Speedrun chart prints Jev at 9.8% on OSWorld-50, 125 seconds a task
This topic was created 28 days ago, and the information it contains may have evolved or changed since then.
Pranjal Aggarwal posted a CUA Speedrun chart on October 2, 2026. The printed callout is Jev at 9.8% success on 50 OSWorld tasks, 125 seconds a task. The chart's own annotation says that is as fast as Astra xhigh at 9x lower accuracy. The opening post says 1.5x slower than Opus-5.5-xHigh and 10x lower accuracy. Those two comparisons are not the same sentence.
Pranjal Aggarwal posted a CUA Speedrun chart on October 2, 2026. The plot is success against wall-clock time on 50 OSWorld tasks. The site is cuaspeedrun.com. The agent code he links is cua-speedrun/cua-speedrun, on the jev-agent tree.
The callout drawn on the chart says Jev accuracy is 9.8% and time per task is 125 seconds. A red note on the same image says that point is as fast as Astra xhigh, at 9x lower accuracy.
The opening of the post describes the run differently. It says the agent is 1.5x slower than Opus-5.5-xHigh, 10x lower in accuracy, and uses 5x more steps, and the author calls the result GPT-4 league on accuracy and Astra-xHigh on time. The chart annotation and that sentence do not name the same reference model, and they do not use the same ratio. Both lines are in the post. The other dots on the scatter plot have no printed coordinates, so this note does not estimate them from where they sit.
The thread separates the model call from the task. A Jev call is under 0.2 seconds, against 5 to 10 seconds for Opus, and the agent takes about 10 times as many steps. Jev does not take the screen as an image. A parser turns the screen into text at about 500 ms a step, against an image prefill the author puts under 200 ms. That is his account of how a fast call still averages 125 seconds a task.
A second picture in the thread is a cartoon. It is not a table. The share image for this page is the scatter plot.
We did not rerun the 50 tasks. The 9.8% figure is the callout on the chart.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Pranjal Aggarwal, @PranjalAggarw16, October 2, 2026, 18:36 UTC, status 2106091065388261599. The chart image is the one saved for this page. Its title is "OSWorld-50 Success Rate vs. Time per Task." The subtitle says average success on 50 OSWorld tasks and average wall-clock time per task.
A callout box on that chart prints Jev accuracy 9.8% and time per task 125 seconds. A red note on the chart reads "as fast as Astra xhigh, 9x lower accuracy." The opening text of the post says the run is 1.5x slower than Opus-5.5-xHigh, 10x lower accuracy, 5x more steps, and in the author's framing a GPT-4 result that matches Astra-xHigh on time.
The thread says a Jev model call is under 0.2 seconds, against 5 to 10 seconds for Opus, and that the agent takes about 10x more steps. It says Jev is not multimodal, so a screen parser adds about 500 ms a step, against an image prefill under 200 ms. The code link is github.com/cua-speedrun/cua-speedrun/tree/jev-agent. The site is cuaspeedrun.com.
Other points on the scatter plot are not labelled with numbers we are willing to read off the axes. A second image in the thread is a cartoon. It is not the table, and it is not the share image. We did not rerun the 50 tasks.
Compare
9.8% and 125 seconds are the only success and time figures printed as a callout. The chart annotation and the opening post name different reference models and different ratios. This note keeps both, and does not pick one.
OSWorld-50 is a computer-use task set. It is not JevBench, and it is not the 60-task computer-use line on the GEV-26B-Decide card. The parser time in the thread is the author's account of why a sub-0.2-second model call still lands at 125 seconds a task.
Terms
- OSWorld-50
- The 50-task slice on the October 2 CUA Speedrun chart. The printed Jev callout is 9.8% success and 125 seconds of wall-clock time per task. Other agents on the plot are not given printed coordinates in this note.
- screen parser
- The thread's account of the extra step in front of text-only Jev. About 500 ms a step, against an image-model prefill the author puts under 200 ms. The model call itself is put under 0.2 seconds.