Organizations are deploying AI-assisted decision tools faster than they are building the human oversight structures those tools require. Automation-related complacency and automation bias, well documented across four decades of human factors research, describe how people, including trained experts, shift from actively verifying automated outputs to passively accepting them once a track record of trust has been established. A separate and equally well-documented body of research on algorithm aversion describes the opposite failure: people abandoning statistically superior algorithmic tools the first time they see one make a visible error, even when the alternative is demonstrably worse human judgment. This article examines what the evidence shows about both failure modes, why standard governance policy addresses neither of them directly, and what functional AI oversight actually requires at the level of decision architecture rather than policy language.
The Oversight Problem Most Governance Policies Miss
Most organizational conversations about AI governance focus on policy: which tools are approved for which use cases, what data can be fed into them, what disclosures are required to employees or customers. These are legitimate governance questions, and most organizations that have thought seriously about AI adoption have already begun answering them. What this policy-first approach consistently misses is the question the research treats as more consequential: what actually happens to human judgment once an AI system becomes a routine part of a decision process, not on the day it is introduced but months or years into its use.
Parasuraman and Manzey's (2010) comprehensive review of automation research in human factors found that people reliably shift from actively verifying automated outputs to passively accepting them, a pattern present in both novice and expert populations and one that is not reliably corrected by training, instruction, or professional experience. Automation complacency, the tendency to reduce active monitoring of a system once it has proven reliable, occurs specifically under conditions of competing task demands, which describes almost every real organizational context in which AI tools are deployed. The manager reviewing an AI-generated performance summary while also managing four other priorities that morning is precisely the condition under which this research predicts oversight degrades.
Automation bias, the related but distinct phenomenon of treating automated recommendations as a heuristic replacement for independent analysis, produces a specific and measurable pattern of error: both omission errors, failing to notice a problem the automated system should have flagged, and commission errors, following an incorrect automated recommendation that a person would not have made independently. Parasuraman and Manzey found this pattern in professional populations, not just laboratory subjects, and found that it was not prevented by prior training or by explicit warnings about the possibility of system error. The implication for organizational governance is direct: a written policy requiring human review of AI-assisted decisions describes an intention, not a safeguard, unless it is paired with structural mechanisms that account for how human vigilance actually degrades over time.
The organizations most exposed to this risk are rarely the ones that adopted AI carelessly. They are the ones that adopted it carefully, built a genuine and well-earned track record of trust in the tool through consistently good early outputs, and are now experiencing exactly the complacency the research predicts as that trust matures from cautious verification into unexamined habit.
The Opposite Failure Is Equally Well Documented
A parallel and, in a business context, equally important body of research documents the reverse problem. Dietvorst, Simmons, and Massey (2015) found that people who observe an algorithm make even a single visible error subsequently lose confidence in it disproportionately to the actual severity of the error, frequently abandoning a tool that was outperforming human judgment on the relevant task before that error occurred. The researchers term this algorithm aversion, and their experimental work found it held even when participants had directly observed the algorithm outperforming a human forecaster across a series of trials.
This finding matters for organizations because it means a single publicized AI failure, the kind that tends to circulate quickly through informal channels regardless of its actual statistical significance, can push an organization toward under-using a genuinely valuable tool just as reliably as unchecked automation bias can push it toward over-using a flawed one. Neither failure mode is a matter of insufficient intelligence or insufficient caution on the part of the people involved; both are well-documented, predictable patterns in how humans process the specific kind of uncertainty that automated and algorithmic tools introduce.
The same research team's follow-up work identified a specific and practically useful mitigation. Dietvorst, Simmons, and Massey (2018) found that people were substantially more willing to continue using an imperfect algorithm, and performed measurably better as a result, when they retained even a small, restricted ability to adjust its outputs rather than being asked to accept the algorithm's recommendation entirely or reject it entirely.
The governance implication is specific: organizations that design AI adoption as a binary choice, either full automation of a decision or full manual control with the AI serving only as an optional reference, are working directly against the mechanism the research identifies as necessary for sustainable, appropriately calibrated trust.
Where This Plays Out Most Visibly: Personnel Decisions
The stakes of both failure modes are highest in exactly the domain where AI adoption in organizations is accelerating fastest: personnel decisions, including hiring, performance evaluation, and promotion recommendations. These are precisely the decisions where automation bias creates the most serious downstream exposure, because an unscrutinized AI recommendation that reflects a biased pattern in historical data does not merely produce one bad decision; it can systematically reproduce that bias across every subsequent decision the tool informs, at a scale and consistency no individual biased human reviewer could match.
This is also the domain where algorithm aversion carries the highest practical cost, because well-designed AI-assisted screening and evaluation tools have shown real potential to reduce certain forms of inconsistency in human judgment, and organizations that abandon such tools entirely after a single visible error may be discarding a genuine improvement over the status quo they are reverting to.
Organizations building AI-assisted personnel processes without structural safeguards are not simply taking on abstract reputational risk. They are operating exactly the conditions under which the research predicts the most consequential and hardest-to-detect failures, because personnel decisions combine high stakes, infrequent individual review of any single decision, and strong incentives for a reviewer under time pressure to defer to a tool that appears to be working.
What Functional Governance Actually Requires
Taken together, this research points toward a small number of concrete structural practices rather than a general organizational commitment to oversight, which the evidence suggests is not sufficient on its own regardless of how sincerely it is held. First, decision authority over each significant AI-assisted use case needs a specifically named owner, not a general policy statement assigning responsibility to a department or role in the abstract.
Second, review processes need built-in structural friction rather than relying on reviewers to independently sustain the vigilance the research shows people cannot reliably maintain on their own once trust has been established. A required documented rationale for accepting or overriding a significant AI-generated recommendation, even a brief one, creates a moment of forced engagement that passive review does not.
Third, the people actually using AI-assisted tools need some genuine, if limited, ability to adjust or override outputs as a standard feature of the process, not only as an emergency escalation path reserved for clear failures. Fourth, leaders need to visibly model calibrated skepticism themselves, treating a single AI error as specific information to be understood rather than grounds for organization-wide retreat, while treating a long, consistent run of correct outputs as a reason to increase verification rigor, not relax it.
Organizations that get AI governance right in practice are rarely distinguished by the length or specificity of their policy documents. They are distinguished by having built decision structures that account for how human judgment actually degrades under conditions of both excessive trust and excessive distrust, and by treating oversight as an ongoing operational discipline rather than a one-time approval decision.
- Dietvorst, B. J., Simmons, J. P., and Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126.
- Dietvorst, B. J., Simmons, J. P., and Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155-1170.
- Parasuraman, R., and Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381-410.