- Due process attaches when AI influences a government decision affecting rights — not only when AI makes the final call. A human who acts on an AI recommendation without genuine independent review is not satisfying the due process obligation.
- Robodebt imposed debt recovery before individual assessment. MiDAS garnished wages before notice. Both violated due process not through AI error alone, but because consequences preceded review. Consequences before assessment is the design failure that creates liability.
- A 90-day backlog for human review is not a meaningful appeal mechanism. A second automated pass of the same algorithm is not an appeal. Challenge mechanisms must be designed for the population that will actually use them — not for applicants with legal representation.
- Automation bias — accepting AI output as correct without independent judgment — is how nominal human review is produced. Track override rates. A 0% override rate is an indicator of automation bias, not model accuracy.
- Notice, explanation, and a genuine path to challenge are not features. They are acceptance criteria. Every jurisdiction with meaningful rule-of-law protections requires all three before AI-influenced government decisions can be legally imposed.
When a government tells someone they owe a debt, their benefits are cut, they are a fraud risk, or they must report for further assessment, that government action can destroy a person’s financial stability, restrict their freedom, or cut off essential support. AI does not change the legal rules that constrain how governments exercise power over individuals. It creates new pressures on them. An automated system can issue thousands of adverse decisions before a human notices a problem, apply a flawed rule consistently across an entire population, and operate in ways that make it genuinely difficult for affected individuals to understand what happened.
What Due Process Requires
Due process is not a single legal standard — it is a principle running through administrative law, constitutional law, and human rights frameworks across democratic systems. Its core elements are consistent: notice (affected persons must know a decision has been made and why), explanation (they must receive the reasons in a form they can understand and challenge), and a genuine opportunity to contest the decision before a decision-maker with authority to change it.
| Jurisdiction | Applicable Framework | Core Requirements for Government AI |
|---|---|---|
| Australia | Administrative Decisions Act, natural justice, post-Robodebt reforms | Reasons for administrative decisions; natural justice obligation; appeal to Administrative Review Tribunal; individual assessment, explanation, and practical appeal path before recovery action. |
| Canada | Federal Court Act, Canadian Bill of Rights, Charter of Rights and Freedoms | Procedural fairness; right to know the case against you; opportunity to respond; Canada’s Directive adds: notice of AI use, plain-language explanation, human review on request by impact level. |
| European Union | ECHR Article 6 (fair trial), GDPR Article 22, EU AI Act Articles 13, 14, 86 | Right not to be subject to solely automated decisions that significantly affect you; right to explanation; human oversight with override capability for high-risk systems; Article 86 right to explanation effective August 2026. |
| Netherlands | Dutch Administrative Law, ECHR Article 8, GDPR | SyRI ruling: AI system that cannot disclose its working to affected persons cannot lawfully operate; proportionality between social interest and privacy intrusion. |
| United Kingdom | Human Rights Act 1998, administrative law, data protection | ECHR Article 8 protection; data protection rights around automated processing; ATRS transparency requirements; rule of law obligations constraining arbitrary state action. |
| United States | Fifth and Fourteenth Amendments, Administrative Procedure Act | Constitutional due process for deprivation of protected interests (benefits, liberty); APA procedural requirements; statutory procedural rights vary by program. |
Four Documented Failures and Their PM Lessons
Robodebt — Consequences Before Assessment
The structural due process failure in Robodebt was consequences-before-assessment: the automated system issued debt notices, and debt recovery proceeded without individual human assessment of whether the debt was real. Affected individuals received letters asserting they owed money, with the obligation to prove they did not. The evidentiary burden was reversed. The 57 Royal Commission recommendations include: governments must not commence or continue recovery action while a debt is disputed. Systems must provide ample and appropriate opportunities to challenge proposed debts before referral for recovery.
PM lesson: Irreversible consequences — debt recovery, benefit termination, reporting to credit agencies, referral to enforcement — must not follow automatically from AI outputs. Human review before consequence is a due process requirement. Design it as a non-negotiable project constraint.
Netherlands SyRI — Opacity as Rights Violation
The due process problem in SyRI was the impossibility of meaningful challenge. SyRI generated risk scores in secret using risk indicators the government refused to disclose. Affected individuals received a risk designation without explanation of what data drove it, what rules were applied, or how they could dispute it. The Dutch government refused to disclose the risk model even to the court. The Hague District Court’s February 2020 ruling found SyRI violated ECHR Article 8 because its operation was “insufficiently clear and verifiable.”
PM lesson: If your government AI system cannot produce a plain-language explanation of how it reached a specific conclusion for a specific individual — one that can be disclosed to that individual and their legal representative — it is not legally defensible in any jurisdiction with meaningful rule-of-law protections.
Michigan MiDAS — The Scale of Systematic Error
MiDAS generated a 93% error rate in some periods from 2013 to 2015, flagging 34,000+ people for unemployment fraud incorrectly. Affected individuals had wages garnished and tax refunds seized before they were meaningfully informed. The appeal process required document submission by mail. Many did not learn of the determination until consequences had already begun.
PM lesson: An appeal mechanism that is technically available but practically inaccessible is not a due process safeguard. Design appeal mechanisms for the populations that will actually use them: accessible in plain language, requiring minimal documentation burden, with timelines that don’t allow harm to compound during the appeal process.
FRT Wrongful Arrests — Automation Bias at the Point of Enforcement
In every documented FRT wrongful arrest in the US — at least seven as of the DOJ’s December 2024 report, almost all involving Black individuals — the pattern is consistent: police treated the facial recognition match as definitive and arrested without conducting corroborating evidence review that would have exonerated the person. The Robert Williams settlement (summer 2024) in Detroit resulted in the strictest FRT operational guidelines in the US: police cannot apply for an arrest warrant based solely on an FRT result.
PM lesson: Automation bias is a human oversight design problem. If your system’s workflow makes it easy to accept the AI recommendation and hard to exercise independent judgment, you have designed automation bias in. Due process-compliant design requires that the human review step is genuinely required, documented, and cannot be bypassed.
Designing Due Process Into Government AI
Notice: Telling People When AI Is Involved
- Notice should identify that AI was used, what type of system, and the general nature of its role (did it generate a risk score? recommend a decision? automate a classification?).
- Notice must be delivered at the point of decision — not buried in a terms and conditions document. For decisions communicated by letter, the letter must include the notice.
- Required under Canada’s Directive at Level 2 and above, EU AI Act Article 13, and implied by due process reasoning in SyRI and post-Robodebt standards.
Explanation: Why the AI Reached This Conclusion
- Explanations must be specific — not generic. “Our system identified you as a higher-risk applicant” does not satisfy the requirement.
- For benefits decisions: the explanation should identify the specific factors the system evaluated and how they contributed to the outcome.
- The explanation must be generated from the model’s actual decision logic, not from a separate explanation system that approximates the model. This is the same standard as the CFPB’s adverse action notice requirement.
- The explanation must be available in the languages reasonably expected among the affected population.
Genuine Human Review: Not Rubber-Stamping
- The human reviewer must have the information needed to exercise independent judgment, including access to the underlying data, not just the AI output.
- Override rates must be tracked as a governance KPI. A system with a 0% override rate either has perfect AI (unlikely) or humans who are not exercising genuine review.
- For high-impact decisions (Canada Level 4, EU AI Act high-risk): the AI should be a recommendation system, not a decision system.
- Review must be documented: what did the reviewer examine, what judgment did they exercise, and what was the basis for following or overriding the AI recommendation?
The Challenge Mechanism: Accessible and Effective
| Design Requirement | Why It Matters |
|---|---|
| Plain language access | Affected individuals must be able to initiate a challenge without legal assistance for routine challenges. |
| Reasonable timeline | Consequences must not compound during the challenge period. Robodebt’s recovery actions during dispute were the primary harm mechanism. Suspend consequences during active challenge. |
| Human reviewer with genuine authority | The challenge reviewer must be capable of changing the decision, not only confirming the algorithm’s output. Re-running the same algorithm is not an appeal. |
| Feedback loop to the system | Challenge outcomes must be tracked and used to identify systematic errors. A high reversal rate on appeal is an early signal of a model problem. |
| Accessibility for vulnerable populations | Many government AI systems affect populations with disabilities, low literacy, limited technology access, or language barriers. The challenge mechanism must be accessible to them specifically. |
EU AI Act Specific Requirements
- Article 13 — Transparency: High-risk AI must provide sufficient information to deployers and affected individuals to understand how the system works. Requires instructions for use, disclosure of intended purpose and accuracy levels.
- Article 14 — Human Oversight: High-risk AI must be designed so that human oversight can be exercised effectively, including the ability to override or disregard AI outputs and to intervene and shut down the system. Directly applicable to the Robodebt and MiDAS failure patterns.
- Article 86 — Right of Explanation (effective August 2, 2026): Individuals subject to decisions based on high-risk AI that significantly affect them have the right to request an explanation of the AI system’s role in the decision.
- GDPR Article 22 (existing): Right not to be subject to solely automated decisions that significantly affect you. This applies now and covers most government AI decisions unless authorized by law.
PM Responsibilities
| Phase | Key Actions |
|---|---|
| In Scope Assessment | Identify what decisions the AI makes or influences, what rights or interests of individuals are affected, and whether those interests are legally protected. Confirm whether GDPR Article 22, Canada’s Directive, or equivalent frameworks apply. |
| Design Phase | Design notice delivery as part of the decision workflow — not as a separate privacy policy update. Design explanation generation before model finalization. Design the challenge mechanism for the affected population. |
| Deployment | Confirm notice, explanation, human review, and challenge mechanisms are all operational before the system makes its first adverse decision. Brief human reviewers on their due process obligations. |
| Post-Deployment | Monitor override rates monthly. Investigate drops toward zero. Track challenge outcomes quarterly. High reversal rates are model performance signals. Review explanation quality periodically with representative affected individuals. |
Right-Sizing for Your Situation
For PMs new to government AI. Covers notice and explanation requirements by jurisdiction, basic human review design, accessible challenge mechanism design, and automation bias mitigation fundamentals.
For PMs building due process compliance programs. Comprehensive notice and explanation implementation, human review workflow design with override tracking, challenge mechanism design for vulnerable populations, multi-jurisdiction due process mapping, and EU AI Act Articles 13, 14, and 86 implementation.
For mature organizations. Enterprise-wide due process compliance across multiple AI systems, appeal mechanism program management, challenge outcome analysis feeding model governance, and judicial review readiness for government AI deployments.
Framework References
Australia Robodebt Royal Commission Report (July 7, 2023) — Recommendations 17.1 and 17.2: individual assessment before consequences; practical appeal paths; human decision before recovery action.
Netherlands District Court of The Hague — SyRI judgment (February 5, 2020) — Established that opacity about how a government AI system works defeats legal accountability; ECHR Article 8 proportionality test.
EU AI Act (Reg. (EU) 2024/1689) — Article 13 (transparency), Article 14 (human oversight for high-risk AI), Article 86 (right of explanation, effective August 2, 2026). GDPR Article 22 — Right not to be subject to solely automated decisions; applies now.
Canada Directive on Automated Decision-Making (Treasury Board, 2019, amended 2023) — Notice, plain-language explanation, human review, and appeal rights by impact level.
DOJ Final Report on AI in Criminal Justice (December 3, 2024) — Documented seven wrongful FRT arrests; best practices requiring corroborating evidence; prohibiting arrest warrant applications based solely on FRT results.
This article is part of AIPMO’s Government series. See also: AI Governance in Government | Procuring AI for Government | Law Enforcement and Criminal Justice AI