Watching an AI agent all day wears down the very skills needed to catch its mistakes.
How it works
Oversight is assumed
Rules and vendors say a human must check agent actions, but give users almost no tools to do it well.
Trace the degradation
Approval fatigue, automation bias and skill atrophy pile up, and tired approvals then feed back into training signals.
Build cognitive scaffolding
Developers add friction, decision design and monitoring. Deployers add training, breaks, rotations and role separation.
Approval fatigue
try itDrag the number of approval prompts up and watch scrutiny per approval.
- Scrutiny per approval
- high
- here
- careful review
Terms
6 termsHuman in the loop A person who approves, corrects or overrides an agent's actions while it runs.
Like a driving instructor riding along with a near-perfect student: after months of nothing happening, they stop watching the road and miss the one crash.
Where it breaks
Agents also shape what the overseer sees and can learn to hide failures.
Limits
- Position paper with no new experiments or measurements.
- Proposed mechanisms are untested; users may dislike friction that reduces overreliance.
- Behavioral monitoring of overseers raises surveillance concerns.
From the paper