Automation bias in agentic AI

The EU AI Act codified awareness of automation bias as one element of human oversight for high-risk AI systems, while a 2025 systematic review synthesized experimental evidence across human-AI decision settings. Partnership on AI then documented the agentic extension: longer workflows, speed, scale, and direct action can erode attention and make nominal human review a bottleneck. IMDA's 2026 agentic-AI framework independently described automation bias as a larger concern with increasingly capable agents and recommended locating significant approval checkpoints and auditing whether oversight remains effective.

Sources: s1, s2, s3, s4

Giving a human an approve button does not guarantee meaningful control. Reviewers can habituate to mostly correct proposals, rush repeated alerts, or lack the time and context to reconstruct a long chain of agent decisions. If approval gates become ceremonial, an agent may modify files, send messages, change records, or initiate transactions despite an error that a nominal human-in-the-loop design was meant to catch. This risk links interface design, permissions, workload, monitoring, training, and accountability. It also explains why blanket approval of every step can be counterproductive: too many low-value interruptions may weaken attention at the steps that matter most.

Sources: s1, s2, s3

A procurement agent prepares dozens of routine purchase actions and occasionally proposes a high-value irreversible transaction. Requiring the same hurried click for every action can produce alert fatigue and automatic acceptance. A risk-calibrated workflow can reserve explicit approval for high-stakes or hard-to-reverse steps, show the evidence and intended effect, let the reviewer override or halt execution, and monitor override rates and response times. These controls may improve engagement, but they do not prove that automation bias has been eliminated.

Sources: s1, s2, s3, s4

Overreliance should be measured rather than inferred from any acceptance of agent output; correct reliance can improve performance. Explanations, transparency, or extra approval prompts can sometimes add cognitive load instead of reducing bias. Evidence from traditional decision-support settings does not transfer automatically to every autonomous workflow, and the agent-specific guidance remains developing. Legal duties under the EU AI Act apply only within the Act's scope, while IMDA's framework is governance guidance rather than a universal legal standard. Controls must be matched to stakes, reversibility, user expertise, workload, and system affordances.

Sources: s1, s2, s3, s4