Published
Updated
Deel's charts show 6 of 8 quality checks at parity, and two losses
This topic was created 31 days ago, and the information it contains may have evolved or changed since then.
TypeSafe posted four charts on September 25 and attributed them to Deel. Six of eight quality checks are at parity or better. Messy metric questions fall 16.6 points. Three-level ticket tags fall 3.6. The live-shadow speed bar is 4 times. The tweet's 50% to 86% and 70% to 97% are not printed on the chart. We did not rerun it.
On September 25 TypeSafe posted four charts and said they came from Deel. The opening card lists four AI use cases, a specialised classifier against what runs today, up to 59 times cheaper, up to 4 times faster, and 6 of 8 quality checks at parity or better. The footer says the comparison is TypeSafe Jev against frontier LLMs, on real Deel data. The thread does not name the frontier model. It does not print how many items sit behind a bar. We did not find a Deel-authored write-up of the same numbers. We did not rerun the calls.
The jobs in the thread are all picks from a known set. Match a repeat analytics question to an approved answer. Pick 1 of 36 metrics. Block PII. Decide when a support chat needs a person. Tag a ticket on a 3-level taxonomy. Sort an expense into about 55 categories.
The quality chart is the place the 6 of 8 comes from. It prints the change in accuracy, in points, against the current approach. Expense categorization is +36. Repeat-question matching is +26.7. Declining when there is no answer is +8.3. Metric pick on well-formed questions is +2. Escalations caught are at parity, with fewer false alarms. The PII guardrail is at parity, and the chart says 100% on both sides. Three-level root-cause tagging is -3.6. Metric pick on messy real questions is -16.6. A footnote says the expense bar is against today’s rule-based receipt matching. The other quality bars are against the frontier LLM on the same task.
The tweet in the same thread says repeat-question matching went from 70% to 97%, and expense categorization from 50% to 86% versus human reviewers. A move from 70 to 97 is 27 points, next to the chart’s +26.7. A move from 50 to 86 is 36 points, the same gap as the expense bar. The chart does not print 70, 97, 50, or 86, and the expense footnote names a rule-based matcher. The tweet names human reviewers.
The cost chart is against the frontier LLM in use today. Metric pick is 59 times. Escalate-to-human is 49 times. Root-cause ticket tagging is about 45 times, and that bar’s footnote says the figure is estimated from per-token pricing. The PII guardrail is 20 times.
The speed chart prints four bars. Live-shadow escalation is 4 times. Metric pick is 2.8 times. Offline escalation is 1.9 times. PII is at parity. The live-shadow footnote says the traffic is thousands of production conversations, run in parallel with the current model. The tweet says offline tests were 2 to 3 times faster. The chart’s offline escalation bar is 1.9 times, and the metric-pick bar is 2.8 times.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
The September 25 thread is from TypeSafe. The opening card says four AI use cases at Deel, a specialised classifier against what runs today, up to 59 times cheaper, up to 4 times faster, and 6 of 8 quality checks at parity or better.
The footer says head-to-head evals on real Deel data, TypeSafe Jev versus frontier LLMs. The jobs named in the thread are repeat-question matching, picking 1 of 36 metrics, blocking PII, deciding when a support chat needs a person, tagging a ticket on a 3-level taxonomy, and sorting an expense into about 55 categories.
The quality chart prints point changes: expense +36, repeat-question matching +26.7, declining when there is no answer +8.3, metric pick on well-formed questions +2, escalations caught at parity with fewer false alarms, PII guardrail at parity (100% both), 3-level root-cause tagging -3.6, metric pick on messy real questions -16.6.
The expense footnote says that bar is against today's rule-based receipt matching. The other quality bars are against the frontier LLM on the same task. The tweet says expense categorization 50% to 86% versus human reviewers, and repeat-question matching 70% to 97%.
Those four percentages are not on the chart. Cost bars against the frontier LLM in use today: metric pick 59 times, escalate-to-human 49 times, root-cause ticket tagging about 45 times (estimated from per-token pricing), PII guardrail 20 times.
Speed bars: live shadow escalation 4 times, metric pick 2.8 times, offline escalation 1.9 times, PII at parity. The live-shadow footnote says thousands of production conversations, run in parallel with the current model. The tweet says offline tests were 2 to 3 times.
The chart's offline and metric bars are 1.9 times and 2.8 times. No denominator, no named frontier model, and no Deel-authored URL. We did not rerun the calls.
Compare
Bryo's 1,565-email table and Kumar's filter are other production measurements on this desk. Neither uses Deel's labels. JevBench p50 is a harness latency, not a shadow of a support queue.
The 4 times bar here is the live-shadow escalation row on the speed chart.
Terms
- 6 of 8
- The opening card's count of quality checks at parity or better. The chart's two losses are messy metric questions, down 16.6 points, and 3-level ticket tags, down 3.6 points.
- 59 times
- The largest cost bar on the chart, metric pick against the frontier LLM in use today. Escalate-to-human is 49 times. Root-cause tagging is about 45 times, estimated from per-token pricing. PII is 20 times.
- live shadow
- The speed chart's footnote for the 4 times escalation bar. Thousands of production conversations, run in parallel with the current model. Offline escalation on the same chart is 1.9 times.