AI in Insurance Claims: Governance for Automation and Denial

Insurance claims processing is where AI governance failures cause the most direct, immediate consumer harm. A wrongful underwriting decision affects future access to coverage. A wrongful claims denial affects a person who has already experienced a loss, is already financially or physically stressed, and is depending on the insurance they paid for to respond. The stakes are high and the timeline is urgent.

The global regulatory and litigation response to claims AI has accelerated sharply in 2024–2025. The Lokken v. UnitedHealth litigation is the most prominent, but it is not isolated. Kelly v. State Farm (Alabama, October 2025) alleged AI was used in claims processing without adequate human review. Multiple Medicare Advantage class actions followed the same pattern. The California SB 1120 prohibition on solely algorithmic coverage denial became effective in January 2025. The EU’s explanation right under Article 86 takes effect for high-risk AI in August 2026. Courts in multiple jurisdictions have begun granting discovery into insurer AI systems.

This article covers the governance obligations for claims AI: human oversight design, explainability requirements, bad faith exposure, the discovery standard, and the PM responsibilities that follow.

The Lokken Discovery Standard and What It Means for All Insurers

The Lokken v. UnitedHealth litigation is a landmark in insurance AI governance for two distinct reasons. First, it allowed the class action to proceed on the theory that algorithmic claims denial at rates inconsistent with individual clinical review constitutes a breach of contract and bad faith. Second, and equally important for all insurers in all lines, a subsequent motion ruling allowed discovery into UnitedHealth’s AI systems — including AI chat files, use policies, and documents concerning AI oversight and governance.

This discovery ruling is broadly applicable. A Minnesota federal judge ruling in a health insurance case establishes a precedent that litigants in auto, property, life, and liability insurance are already citing. The message is unambiguous: AI decision-making in insurance claims is subject to discovery. Insurers that cannot produce — in response to a discovery request — documentation of how their AI works, what it was tested for, what governance was applied, and what human oversight existed, are in a materially worse litigation position than those who can.

California SB 1120: The Legislative Standard

California SB 1120, effective January 1, 2025, makes California the first US jurisdiction to legislatively prohibit health insurers from denying, delaying, or modifying coverage for medically necessary treatment based solely on an algorithm or automated tool. Key provisions:

• Any adverse benefit determination that affects a patient’s care must be individually reviewed by a licensed clinician with appropriate expertise for the clinical question.
• The clinician reviewer must review the actual clinical record, not just the AI output. A review that consists of approving an AI recommendation without reviewing the underlying clinical documentation does not satisfy SB 1120.
• The denial must be based on the individual patient’s clinical circumstances, not solely on population-level statistical outputs from an AI tool.
• The law does not prohibit AI use in claims processing — it prohibits sole reliance on AI for adverse benefit determinations. AI can assist clinical review; it cannot replace it for covered health services.

SB 1120 represents the legislative codification of the standard courts applied in Lokken: AI cannot substitute for individual clinical assessment in decisions about whether a patient receives covered care. Other states are tracking this legislation. The American Hospital Association has proposed similar requirements nationally. For insurers operating in California and planning for national compliance evolution, SB 1120 is the forward-looking standard.

EU Requirements: Explanation Rights and Redress Mechanisms

For EU insurers deploying AI in claims processing, two distinct obligations apply:

• GDPR Article 22 (currently in force): Right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects. Insurance claims denials clearly qualify. Insurers must provide human review on request, an explanation of the automated decision, and a mechanism to contest it. This is an existing legal obligation, not a future compliance item.
• EU AI Act Article 86 (effective August 2, 2026): Individuals subject to decisions based on high-risk AI that significantly affect them have the right to request a plain-language explanation of the AI’s role, the main factors considered, and the impact on the decision. For insurance AI classified as high-risk — life and health insurance risk assessment and coverage decisions under Annex III — this is a binding obligation requiring the AI system to produce individual explanations on request.
• EIOPA Opinion (August 2025): Requires adequate redress mechanisms for customers negatively affected by AI systems. This applies to all insurance AI in the EU/EEA, not just high-risk AI. A claims AI system that generates denials without a practical, accessible appeal mechanism to a human reviewer does not satisfy EIOPA’s requirements.

The Bad Faith Framework

In insurance law, bad faith occurs when an insurer unreasonably denies or delays a legitimate claim. AI-driven claims denial creates bad faith exposure in several ways:

• Sole reliance on AI without human review: An AI-generated denial with no meaningful human clinical review may be evidence of unreasonable claim handling. If the denial was based on an AI output that a human reviewer with appropriate expertise would have rejected, the failure to conduct genuine review may constitute bad faith.
• AI hallucination in claims processing: An insurer that erroneously and unreasonably denies a claim on the basis of an AI output that contains a factual error or hallucination, with no human verification of its reasonableness, faces bad faith exposure. If denial is based on an AI hallucination not caught by a human, the policyholder can use that to argue the insurer acted in bad faith.
• Systematic denial pattern: The Lokken theory applies beyond health insurance. Where AI generates denials at rates that depart significantly from what human review would produce, the differential is evidence that the AI is doing something different from genuine claims assessment. A pattern of AI denials that are reversed on appeal — or would be if challenged — is evidence of systematic bad faith.
• Inadequate explanation: An AI-generated denial that does not provide a specific explanation of why the claim was denied, sufficient for the policyholder to understand and challenge the decision, may fail the duty to communicate fairly. This obligation is independent of the AI — it has always existed; AI creates a new way to fail it at scale.

Governance Design for Claims AI

Human Oversight: Genuine, Not Nominal

The consistent failure pattern across Lokken, California SB 1120, EIOPA’s Opinion, and the EU AI Act is the same: AI claims denial without genuine human review. Designing genuine human review requires:

• The reviewer must be qualified for the clinical or technical question the claim raises. A non-specialist approving an AI denial of a complex oncology coverage claim does not constitute genuine review.
• The reviewer must have access to the underlying claim record, not just the AI output. Review limited to “does this AI recommendation look reasonable” is not individual claim assessment.
• Override authority must be real and explicitly expected. A review process that produces 0% overrides of AI denials is not a genuine review process. Track override rates; investigate declines toward zero.
• Document the review: what the reviewer examined, what judgment was exercised, and the basis for following or overriding the AI recommendation. Documentation of genuine review is both governance evidence and the discovery record.

The Explanation Stack

Claims AI must be able to produce explanations at three levels:

• Regulatory level: Technical documentation of model architecture, training data, performance metrics, bias testing results — for regulator examination. Kept internally, produced for market conduct examination.
• Legal level: Documentation of specific claim handling decisions — what AI system was used, what input it received, what output it produced, and what human review was conducted. This is the discovery record. Maintain for the litigation limitation period applicable to the claim type.
• Consumer level: Plain-language explanation for the policyholder of why the claim was denied, what factors the AI considered, and how to contest the decision. Required by California SB 1120, GDPR Article 22, EU AI Act Article 86 (2026), and EIOPA’s Opinion. Must be specific enough for the policyholder to formulate an appeal.

AI-Generated Denial Rate Monitoring

Appeal Mechanism Design

• The appeal mechanism must reach a human reviewer with authority and qualification to overturn the AI-generated denial. A second automated AI review is not an appeal.
• Appeal timelines must be practical relative to the urgency of the underlying claim. Health claims have urgency timelines; property claims have repair urgency; liability claims have settlement pressure. Design appeal SLAs accordingly.
• Appeal outcomes must feed back into AI model governance. A pattern of reversals on specific claim types or denial reasons is a model performance signal, not just a customer service issue.

PM Responsibilities for Claims AI

• Before deployment: establish the human oversight workflow before the AI goes live. Define who reviews what, with what qualifications, within what timeframes, with what documentation. This is a non-negotiable pre-condition, not a post-launch optimization.
• Before deployment: confirm the explanation capability produces specific, accurate, comprehensible explanations for adverse decisions. If the model cannot do this, it does not meet California SB 1120, GDPR Article 22, or EU AI Act Article 86 standard (when effective).
• At deployment: establish baseline AI-to-human denial rate differential for each major claim category. This baseline is the governance reference for ongoing monitoring.
• Post-deployment: monitor denial rate differentials, appeal reversal rates, and override rates monthly. Investigate alerts. Document findings and remediation. This documentation will be discoverable if there is litigation about claims handling.

Right-Sizing Your AI Governance Approach

The AI Governance Advisor can help you design a human oversight framework and explanation stack for your claims AI — start with a free Essential account.

Framework References

Lokken v. UnitedHealth Group (D. Minn., February 2025) — AI claims denial class action allowed to proceed; discovery into AI use granted; applicable to all insurance lines. Establishes documentation and bad faith standards for AI claims processing.

California SB 1120 (effective January 1, 2025) — First US prohibition on solely algorithmic coverage denial for medically necessary health care; licensed clinician review required for all adverse benefit determinations.

GDPR Article 22 (EU/EEA, currently in force) — Right not to be subject to solely automated significant decisions; human review on request; explanation obligation; contestability right for insurance claims.

EU AI Act (Reg. (EU) 2024/1689) Article 86 (effective August 2, 2026) — Right to plain-language explanation of high-risk AI’s role in significant decisions; applicable to high-risk insurance AI claims systems.

EIOPA Opinion on AI Governance and Risk Management (August 6, 2025) — Redress mechanisms required for all insurance AI (not just high-risk); explainability to supervisors and clients; ultimate insurer accountability.

NAIC Model Bulletin: Use of Artificial Intelligence Systems by Insurers (December 2023) — Adverse outcome tracking; documentation of AI role in claim decisions; consumer notice requirements.

This article is part of AIPMO’s Insurance series. See also: AI Governance in Insurance  |  AI in Insurance Underwriting  |  Algorithmic Bias in Insurance AI  |  GenAI in Insurance