LiteLLM proxies Jev's evaluate endpoint and uses it to drop stale tool results

LiteLLM posted on September 19 that Jev is supported on the proxy, and that the team already uses it internally. The post points at a compaction write-up.

Kerry Lu’s pass-through note, dated September 20, is the listing. TypeSafe Jev lands in v1.103.0-rc. Jev answers on /v1/systemone, and LiteLLM proxies that path with logging and cost tracking. Set TYPESAFE_API_KEY on the proxy. Clients send a LiteLLM virtual key to LITELLM_PROXY_BASE_URL/typesafe. The sample is POST /typesafe/v1/systemone with state, model: jev-latest, and a choice question. The response is TypeSafe’s, unchanged. Any path under /typesafe/ is forwarded, so GET /typesafe/v1/models lists what TypeSafe lists. Spend is logged under the versioned model TypeSafe reports, typesafe/jev-1.13.0 on that page, even when the request asks for jev-latest. Price is the public $0.042 per million input tokens, no output charge.

The compaction post is the internal use. LiteLLM asks Jev, for each older completed tool exchange, whether the bot still needs that result to answer the latest user question. Default relevance_threshold is 0.2. Results below it are replaced with a one-line notice that the result was removed. Kept results stay as they were. Tool calls and their IDs stay, so the conversation still has the shape the generator expects. System and user messages are left alone. The last assistant message and any tool exchange it belongs to are protected.

The guardrail is pre_call, named typesafe, and works with Chat Completions, Anthropic Messages, and the Responses API. A weather-then-shop-hours sample in the post is the worked example: the user asks when the shop closes, the weather report is long enough to be eligible, and Jev can mark it for removal. The post says turning compaction on does not guarantee a smaller request. If Jev is down, fail_open sends the original messages through. Applied runs can show exchanges_evaluated, exchanges_dropped, chars_removed, and the Jev model id. There is no published savings table.

The earlier compaction write-up on this desk is Tran’s keep-or-drop sieve. AstroHan’s 30-task run is the one that kept the agent and the model fixed and reported task pass rates. We did not run LiteLLM’s proxy.