Most organizational conversations about AI governance focus on policy: which tools are approved for which use cases, what data can be fed into them, what disclosures are required to employees or customers. These are legitimate governance questions, and most organizations that have thought seriously about AI adoption have already begun answering them. What this policy-first approach consistently misses is the question the research treats as more consequential: what actually happens to human judgment once an AI system becomes a routine part of a decision process, not on the day it is introduced but months or years into its use.
Parasuraman and Manzey's (2010) comprehensive review of automation research in human factors found that people reliably shift from actively verifying automated outputs to passively accepting them, a pattern present in both novice and expert populations and one that is not reliably corrected by training, instruction, or professional experience. Automation complacency, the tendency to reduce active monitoring of a system once it has proven reliable, occurs specifically under conditions of competing task demands, which describes almost every real organizational context in which AI tools are deployed. The manager reviewing an AI-generated performance summary while also managing four other priorities that morning is precisely the condition under which this research predicts oversight degrades.
Automation bias, the related but distinct phenomenon of treating automated recommendations as a heuristic replacement for independent analysis, produces a specific and measurable pattern of error: both omission errors, failing to notice a problem the automated system should have flagged, and commission errors, following an incorrect automated recommendation that a person would not have made independently. Parasuraman and Manzey found this pattern in professional populations, not just laboratory subjects, and found that it was not prevented by prior training or by explicit warnings about the possibility of system error. The implication for organizational governance is direct: a written policy requiring human review of AI-assisted decisions describes an intention, not a safeguard, unless it is paired with structural mechanisms that account for how human vigilance actually degrades over time.
The organizations most exposed to this risk are rarely the ones that adopted AI carelessly. They are the ones that adopted it carefully, built a genuine and well-earned track record of trust in the tool through consistently good early outputs, and are now experiencing exactly the complacency the research predicts as that trust matures from cautious verification into unexamined habit.
A parallel and, in a business context, equally important body of research documents the reverse problem. Dietvorst, Simmons, and Massey (2015) found that people who observe an algorithm make even a single visible error subsequently lose confidence in it disproportionately to the actual severity of the error, frequently abandoning a tool that was outperforming human judgment on the relevant task before that error occurred. The researchers term this algorithm aversion, and their experimental work found it held even when participants had directly observed the algorithm outperforming a human forecaster across a series of trials.
This finding matters for organizations because it means a single publicized AI failure, the kind that tends to circulate quickly through informal channels regardless of its actual statistical significance, can push an organization toward under-using a genuinely valuable tool just as reliably as unchecked automation bias can push it toward over-using a flawed one. Neither failure mode is a matter of insufficient intelligence or insufficient caution on the part of the people involved; both are well-documented, predictable patterns in how humans process the specific kind of uncertainty that automated and algorithmic tools introduce.
The same research team's follow-up work identified a specific and practically useful mitigation. Dietvorst, Simmons, and Massey (2018) found that people were substantially more willing to continue using an imperfect algorithm, and performed measurably better as a result, when they retained even a small, restricted ability to adjust its outputs rather than being asked to accept the algorithm's recommendation entirely or reject it entirely.
The governance implication is specific: organizations that design AI adoption as a binary choice, either full automation of a decision or full manual control with the AI serving only as an optional reference, are working directly against the mechanism the research identifies as necessary for sustainable, appropriately calibrated trust.
| Safeguard | Typical stated policy | Typical actual practice |
|---|---|---|
| Human review requirement | Required before acting on AI output | Reduces to passive acceptance as trust builds |
| Named decision owner | Assigned to a department or role | Diffuse; no individual owns a specific use case |
| Override ability | Framed as an exception process | Rarely used; treated as an escalation, not a default feature |
| Response to visible error | Case-by-case review | Often disproportionate abandonment of an otherwise net-beneficial tool |
The stakes of both failure modes are highest in exactly the domain where AI adoption in organizations is accelerating fastest: personnel decisions, including hiring, performance evaluation, and promotion recommendations. These are precisely the decisions where automation bias creates the most serious downstream exposure, because an unscrutinized AI recommendation that reflects a biased pattern in historical data does not merely produce one bad decision; it can systematically reproduce that bias across every subsequent decision the tool informs, at a scale and consistency no individual biased human reviewer could match.
This is also the domain where algorithm aversion carries the highest practical cost, because well-designed AI-assisted screening and evaluation tools have shown real potential to reduce certain forms of inconsistency in human judgment, and organizations that abandon such tools entirely after a single visible error may be discarding a genuine improvement over the status quo they are reverting to.
Organizations building AI-assisted personnel processes without structural safeguards are not simply taking on abstract reputational risk. They are operating exactly the conditions under which the research predicts the most consequential and hardest-to-detect failures, because personnel decisions combine high stakes, infrequent individual review of any single decision, and strong incentives for a reviewer under time pressure to defer to a tool that appears to be working.
Taken together, this research points toward a small number of concrete structural practices rather than a general organizational commitment to oversight, which the evidence suggests is not sufficient on its own regardless of how sincerely it is held. First, decision authority over each significant AI-assisted use case needs a specifically named owner, not a general policy statement assigning responsibility to a department or role in the abstract.
Second, review processes need built-in structural friction rather than relying on reviewers to independently sustain the vigilance the research shows people cannot reliably maintain on their own once trust has been established. A required documented rationale for accepting or overriding a significant AI-generated recommendation, even a brief one, creates a moment of forced engagement that passive review does not.
Third, the people actually using AI-assisted tools need some genuine, if limited, ability to adjust or override outputs as a standard feature of the process, not only as an emergency escalation path reserved for clear failures. Fourth, leaders need to visibly model calibrated skepticism themselves, treating a single AI error as specific information to be understood rather than grounds for organization-wide retreat, while treating a long, consistent run of correct outputs as a reason to increase verification rigor, not relax it.
Organizations that get AI governance right in practice are rarely distinguished by the length or specificity of their policy documents. They are distinguished by having built decision structures that account for how human judgment actually degrades under conditions of both excessive trust and excessive distrust, and by treating oversight as an ongoing operational discipline rather than a one-time approval decision.