Feed
ORCA-bench: How Ready Are Language Model Agents for Oncall?
Top AI coding agents diagnose under a third of realistic production outages correctly.
25.3%
Best RCA accuracy, Medium tasks
Even the best of five frontier agents misses most realistic incidents.
How it works
Oncall is different
Root cause analysis starts from vague complaints, hours late, across noisy metrics, logs, traces and code. Existing benchmarks drop key pieces.
Record a real shop
Six days of telemetry from a 19-service demo store, faults triggered by feature flags, 1,079 tasks with varied report vagueness and delay.
Agents fall short
Best accuracy is 25.3% on Medium and 10.0% on Hard; removing source code lowers accuracy and raises hallucination for every model.
From the paper