- Clinical AI that influences diagnosis, treatment, or care decisions is a regulated medical device in the US, Canada, Australia, and the EU. Deploying it as a ‘productivity tool’ to avoid that classification is a regulatory and liability risk — not a governance shortcut.
- AI scribes report 1–3% hallucination rates. The NHS Anima incident in July 2025 — a healthy patient falsely diagnosed with diabetes and heart disease, triggering downstream clinical invitations — shows what a single unreviewed error cascades into. Clinician review before every AI-generated entry into the patient record is not optional.
- AHPRA, GMC, and the Canadian CMPA all confirmed in 2024–2025 that practitioners remain ultimately responsible for AI-assisted clinical decisions. This is not a liability gap to manage around — it is a design requirement. Clinical AI must be built for clinicians to review, understand, and override.
- Over half of published clinical AI models use training data from the US or China. Health Canada’s February 2025 guidance requires data ‘adequately representative of the Canadian population’ including skin pigmentation and biological sex. Population representativeness is a patient safety issue, not a fairness aspiration.
- EU AI Act Article 10 requires medical device AI manufacturers to address biases ‘likely to affect the health and safety of persons.’ This is active compliance work before August 2027 — and it applies to any developer selling into the EU market, regardless of where they are based.
Healthcare AI is not a single governance problem. It is several distinct problems that interact: the clinical validation problem (has this AI been tested in a population like mine, under conditions like mine?); the algorithmic bias problem (does this AI perform worse for my patients from underrepresented groups?); the ambient AI documentation problem (when AI writes the clinical record, who is responsible for what it says?); and the coverage and access problem (when AI drives decisions about what care patients receive, who is accountable?).
Where AI Is Used in Healthcare
| Function | AI Applications |
|---|---|
| Diagnostic imaging | Radiology AI for detecting cancer, fractures, pathologies; dermatology AI for skin lesion classification; ophthalmology AI for retinal disease screening. |
| Clinical decision support | Sepsis prediction, deterioration alerts, diagnosis suggestion, drug interaction screening, surgical risk scoring. |
| Documentation | Ambient AI scribes generating clinical notes from consultation audio; summarization of patient records; discharge summary generation. |
| Care management and triage | Predictive risk stratification for high-cost utilization, readmission prediction, ED triage support. |
| Coverage and insurance | Claims processing automation, prior authorization AI, coverage denial algorithms. |
| Pathology and genomics | Digital pathology image analysis, genomic variant interpretation, biomarker identification. |
| Mental health | Conversational AI for mental health support, mood tracking, crisis detection. |
Four Cases Across Four Jurisdictions
UK: NHS Anima Health AI Scribe Hallucination (July 2025)
A healthy London patient received an NHS letter inviting him to diabetic eye screening. He had never been diagnosed with diabetes. His GP practice had been using Anima Health’s AI tool Annie, which generated a fabricated medical summary asserting he had Type 2 diabetes and suspected coronary artery disease, along with a fabricated hospital address. A clinician spotted the error but was distracted and saved the original erroneous version. The error triggered the automated downstream process that invites diabetic patients for annual eye screening.
PM lesson: The incident illustrates three independent governance failures: hallucination (AI generated false clinical content), human oversight failure (the reviewing clinician failed to complete the review), and cascade (the error propagated into downstream clinical processes). Governance must address all three independently.
US: Sharp Healthcare / Abridge Consent Lawsuit (December 2025)
Sharp Healthcare was sued for recording patient consultations using Abridge’s AI scribe without adequate patient consent, alleged to violate California’s Confidentiality of Medical Information Act and state wiretapping laws. The same legal theory successfully applied in Turner v. Nuance (financial services). California’s two-party consent requirement was the central issue.
PM lesson: In two-party consent jurisdictions — California and approximately a dozen US states, most of Canada — a deployment process that notifies patients does not necessarily constitute consent. The consent framework must be designed specifically for ambient AI.
US: Lokken v. UnitedHealth (February 2025) and Coverage Denial AI
A federal court allowed claims to proceed against UnitedHealth for using an AI algorithm (nH Predict) to systematically deny post-acute care coverage for elderly patients. The algorithm denied claims at rates that the lawsuit alleged far exceeded what individual human review would have denied.
PM lesson: Insurance and managed care AI that generates coverage denials is clinical AI — not administrative AI. Decisions about what care a patient receives are health decisions with health consequences. Human review of AI coverage denials must be genuine, not nominal.
US: DOJ / Troy Health AI Fraud Enforcement (August 2025)
The DOJ brought enforcement action against Troy Health for generating fraudulent clinical documentation, treatment recommendations, and billing submissions using AI. The case established a prosecutorial standard: AI-generated clinical content not reviewed and verified by a responsible clinician is not just a quality problem. When it produces fraudulent billing or unsupported clinical justifications at scale, it is a federal crime.
The Regulatory Landscape: Five Jurisdictions
| Jurisdiction | Framework | Key Requirements |
|---|---|---|
| United States | FDA SaMD regulation; January 2025 draft guidance | 1,016+ AI/ML-enabled medical devices authorized as of March 2025. Clinical AI intended to replace or supplement clinical judgment is regulated as software as a medical device (SaMD). Predetermined Change Control Plans (PCCPs) for adaptive AI. Demographic subgroup performance testing required. |
| United Kingdom | MHRA, NHS DCB0129/DCB0160 | Ambient scribing tools that give a medical device purpose require MHRA classification. NHS AVT Supplier Registry requires Class 1 MHRA accreditation. NHS AI Airlock programme (launched October 2025) provides regulatory sandbox. DCB0129 (supplier) and DCB0160 (organizational clinical risk assessment) apply before go-live. |
| Canada | Health Canada MLMD Guidance (February 2025) | Four-class risk framework. Requires: explicit disclosure of ML use, clinical evidence for Class II–IV devices, PCCPs for adaptive AI, post-market monitoring. Training data must be ‘justified as adequately representative of the Canadian population’ including skin pigmentation, biological sex. |
| Australia | TGA SaMD regulation; grace period ended November 2024 | AI clinical note tools with diagnostic features may require ARTG registration. TGA July 2025 outcome report: enforcement stance on unapproved AI medical devices. ACSQHC August 2025 guides emphasize assessing AI for potential biases. |
| European Union | MDR/IVDR + EU AI Act (MDAI full compliance August 2027) | Medical device AI under MDR/IVDR + EU AI Act. CE marking required. Article 10: address biases ‘likely to affect the health and safety of persons.’ MDCG 2025-6 (June 2025) for MDR/IVDR interplay. |
Five-Step Governance Framework
Step 1: Regulatory Classification
| Jurisdiction | Classification Framework | Key Trigger |
|---|---|---|
| US | FDA SaMD under 21st Century Cures Act | AI intended to support clinical decisions; intended use determines class. |
| UK | MHRA medical devices under UK MDR 2002 | Outputs relied upon to inform care decisions trigger device classification. |
| Canada | MLMD under Food and Drugs Act; Class I–IV | AI supporting clinical decisions; Class based on risk. |
| Australia | TGA SaMD under Therapeutic Goods Act | SaMD grace period ended November 2024. |
| EU | MDR/IVDR + EU AI Act high-risk AI | Notified Body assessment; CE marking; full MDAI compliance August 2027. |
Step 2: Clinical Evidence and Validation
- Pre-market validation must demonstrate safety and effectiveness in a clinical population representative of the intended deployment population — not just the training population.
- For diagnostic AI: sensitivity, specificity, and AUC by demographic subgroup. Aggregate performance masking subgroup disparities is a regulatory and patient safety gap.
- Independent validation by a party without a financial interest in the AI’s approval is required for higher-risk clinical AI.
Step 3: Human Oversight Design
- For clinical decision support: the AI output is a recommendation, not a decision. The clinician makes and documents the clinical decision independently.
- For AI documentation: clinicians must review and verify all AI-generated content before it enters the patient record. Every AI-generated note should require explicit clinician attestation before completion.
- For coverage denial AI: meaningful human clinical review before any denial that overrides a treating clinician’s recommendation. AI-generated denial rates should be tracked against clinical reviewer override rates.
Step 4: Consent and Transparency
- Patients must know that AI is used in their care — specifically enough to understand what AI does, not just that AI exists.
- For ambient AI documentation: recording patient consultations requires consent in two-party consent jurisdictions. Transmission to third-party AI processors triggers additional data protection obligations.
- Design the consent workflow specifically for ambient AI — not generic privacy policy acknowledgment.
Step 5: Bias Testing and Equity
- Demographic subgroup performance analysis required before deployment for any clinical AI that affects patient care decisions.
- For training data: assess representativeness against the intended patient population. Document what demographic groups are over- and under-represented.
- Bias testing at deployment is a gate, not a one-time exercise. Demographic performance must be tracked in production.
Right-Sizing for Your Situation
For PMs new to healthcare AI governance. Covers regulatory classification basics across US, UK, Canada, Australia, and EU; minimum clinical validation requirements; consent essentials for clinical AI and ambient scribes; and human oversight design fundamentals.
For PMs building systematic healthcare AI governance programs. Comprehensive jurisdiction-by-jurisdiction regulatory mapping, clinical validation methodology, demographic bias testing frameworks, ambient AI consent workflow design, human oversight program design, and post-market surveillance program setup.
For PMs in mature healthcare organizations. Enterprise-wide clinical AI governance, EU AI Act MDAI compliance roadmap (August 2027), multi-jurisdiction regulatory strategy, AI-assisted clinical decision liability framework, and coverage denial AI governance.
Framework References
FDA Draft Guidance: AI-Enabled Device Software Functions Lifecycle Management and Marketing Submission Recommendations (January 6, 2025) — Lifecycle approach; Predetermined Change Control Plans; post-market monitoring obligations for adaptive AI.
Health Canada Pre-Market Guidance for Machine Learning-Enabled Medical Devices (February 2025) — MLMD risk classification, PCCP framework, representativeness requirement including demographic diversity.
EU AI Act (Reg. (EU) 2024/1689) — Article 10 (data governance and bias for MDAI), Article 13 (transparency), Article 14 (human oversight). Full MDAI compliance August 2, 2027.
Australia TGA — Clarifying and Strengthening the Regulation of Medical Device Software including AI (July 2025) — Reclassification recommendations; enforcement stance on unapproved AI medical devices.
AHPRA Guidance on AI in Clinical Practice (August 2024) — Practitioners are ultimately responsible for any AI used in their clinical practice.
NHS England Guidance on AI-Enabled Ambient Scribing Products (2025) — DCB0129, DCB0160, DPIA, Clinical Safety Officer requirements, NHS AVT Supplier Registry.
Joint Commission / CHAI Responsible AI Adoption Guidance (September 2025) — Clinical AI governance best practices; clinical validation standards; deployment governance for health systems.
NIST AI RMF 1.0 — MEASURE 2.11 (demographic subgroup bias testing), GOVERN 1.7 (GenAI-specific governance processes), MANAGE 4.1 (continuous post-deployment monitoring).
This article is part of AIPMO’s Healthcare series. See also: Clinical Validation of Healthcare AI | Algorithmic Bias in Clinical AI | Ambient AI and Consent in Healthcare