Published
AgentOS posts a 121-turn routing table for an opt-in Jev strategy
AgentOS added Jev as a third Pilot Router strategy, off by default. On the project's 121-turn labeled set the chart shows Jev at accuracy 0.901 against the local MiniLM classifier at 0.355. The same chart prices 121 cases at about $0.005 and a call at about 0.9 seconds. A follow-up pull request the same day says it sharpens the R2 and R3 criteria.
AgentOS posted on September 22 that Pilot Router now has a third way to pick a model tier. The existing two are a local MiniLM classifier, which stays the default and does not leave the machine, and an LLM judge that pins its confidence at 1.0. The new one sends the turn to Jev. It is opt-in, marked experimental, and the announcement says the turn text goes to typesafe.ai.
The pull request is use-agent-os/agent-os#3316, opened by GitHub user andreapn and merged the same day. One POST /v1/systemone asks two questions. route is a Choice over R0, R1, R2, and R3. high_risk is a Noul. At or above 0.7, code floors the turn at c3, so a short request to delete something in production does not depend on the Choice reading that sentence as deep work. A missing key, an HTTP error, a timeout, or a bad body degrades to jev_unavailable. There is no retry inside the router budget. The API key is taken from TYPESAFE_API_KEY.
The numbers are on a chart in the second post of the thread. The footer says the corpus is tests/data/router_eval/cases.jsonl, 121 turns, gold labels R0 to R3, Vietnamese, Chinese, and English, classifier only, and that two repeat runs were identical. Local MiniLM against Jev:
| Metric | Local | Jev |
|---|---|---|
| Accuracy | 0.355 | 0.901 |
| Under-routing | 0.413 | 0.066 |
| Over-routing | 0.231 | 0.033 |
| Macro-F1 | 0.894 | |
| R3 recall | 0.907 | |
| English | 0.513 | 0.946 |
| Vietnamese | 0.312 | 0.938 |
| Chinese | 0.250 | 0.806 |
| Calibration ECE | 0.062 |
The chart prices the local classifier at no network and no money, and Jev at one call per turn, about 0.9 seconds, with 121 cases at about $0.005. Chinese is the low slice on that chart, 0.806 against 0.946 in English. The pull request also describes a smaller live smoke: three Vietnamese turns landed on R0 or R1, with confidence from 0.21 to 0.98. That is three turns, not the 121.
A second pull request, #3317, merged the same day. Its title says it sharpens the Jev criteria for R2 and R3. The chart above is the one in the announcement thread. We did not run the eval script, and we did not read a results file attached to 3317. The account says the local classifier remains the default. TypeSafe’s Master Customer Agreement section 2.3(f) forbids customers from publishing benchmarks of the Services.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Thread from @useAgentOS, September 22, 2026, 06:28 UTC. Pull request 3316 by GitHub user andreapn, merged the same day, commit 880bb9a. The strategy is named jev, labelled experimental in the selectors, and is not the default. It POSTs the current turn, head and tail truncated, to typesafe.ai /v1/systemone. Two questions: a Choice named route over R0 to R3, and a Noul named high_risk. At or above high_risk_threshold, default 0.7, code floors the turn at c3. Missing key, HTTP errors, timeouts, and a malformed body degrade to routing_source jev_unavailable with no in-band retry inside the router budget. The key is read from TYPESAFE_API_KEY and is not written to config.toml when it matches the environment variable. The chart in https://x.com/useAgentOS/status/2102283783580049462 is titled Pilot Router, local ML vs Jev. Footer: corpus tests/data/router_eval/cases.jsonl, gold labels R0 to R3, two repeat runs identical, classifier only, Vietnamese, Chinese, and English. Local MiniLM (pilot-v1) versus Jev: accuracy 0.355 / 0.901, under-routing 0.413 / 0.066, over-routing 0.231 / 0.033, macro-F1 blank / 0.894, R3 recall blank / 0.907, English 0.513 / 0.946, Vietnamese 0.312 / 0.938, Chinese 0.250 / 0.806, calibration ECE blank / 0.062. Notes on the chart: local is 0 network and $0; Jev is one API call per turn at about 0.9 s; 121 cases about $0.005. The chart says Jev sends turn text off the device, opt-in, experimental, and that local stays the default. The pull request's live smoke is a different sample: three Vietnamese turns landed R0 or R1 at confidence 0.21 to 0.98, which the author ties to a TypeSafe note that the model is less optimized for non-English. Pull request 3317, also merged September 22, is titled as a sharpening of the Jev R2 and R3 criteria. The chart is the one in the announcement thread, not a report attached to 3317. We did not run evaluate_jev.py. TypeSafe's Master Customer Agreement section 2.3(f) forbids publishing benchmarks of the Services. The figures are reported as published.
Compare
Vercel's form router keeps one destination at confidence 0.95 on a single submission. semgate asks one question about an HTTP request. jev-gateway picks a tool and can skip the language model. This table is a tier label, R0 to R3, on 121 turns the project already labeled, against its own on-device MiniLM classifier. It is not a JevBench score. The local column leaves macro-F1, R3 recall, and ECE blank.
Terms
- Pilot Router
- AgentOS's layer that picks a model tier for a turn, from c0 for a short reply to c3 for deep work. The Jev strategy is opt-in. The default remains the local MiniLM classifier.
- high_risk floor
- In the AgentOS Jev strategy, a Noul at or above 0.7 forces the turn to tier c3 in code, so a short request to delete production data does not depend on the route Choice reading the words literally.