NEW - AI in Insurance: A PM's Guide to a High-Stakes Sector

Insurance is one of the most AI-intensive industries — and one of the most heavily regulated. AI systems are making decisions about who gets coverage, how much they pay, and whether their claims get approved. When those decisions go wrong, the consequences are severe: denied care, discriminatory pricing, regulatory action, and class-action lawsuits.

If you're managing AI projects in insurance, you're operating in a sector where the governance stakes are exceptionally high. The frameworks that apply — from the EU AI Act to the NAIC Model Bulletin to NIST AI RMF — are specific, detailed, and enforceable. This article translates them into PM terms. 

Where AI Is Used in Insurance

AI touches nearly every stage of the insurance lifecycle.

NAIC surveys of insurers found high AI adoption rates across every major line: 92% of health insurers, 88% of auto insurers, 70% of home insurers, and 58% of life insurers report current or planned AI usage. The governance burden is proportional to that adoption. 

The Risks Are Real: Three Cautionary Tales

UnitedHealthcare and the nH Predict Algorithm

In late 2023, families of deceased Medicare Advantage patients filed a class-action lawsuit against UnitedHealth Group, alleging the company used an AI algorithm called nH Predict to wrongfully deny post-acute care coverage. According to a Senate Permanent Subcommittee investigation, UnitedHealthcare's denial rate for post-acute care more than doubled — from 10.9% in 2020 to 22.7% in 2022 — after implementing the algorithm. Approximately 90% of those denials were overturned on appeal by administrative law judges, suggesting the algorithm was systematically wrong.

PM lesson: An AI system that saves money by denying legitimate claims isn’t a success — it’s a liability. Appeal and overturn rates are governance KPIs. Define acceptable thresholds before deployment and track them continuously.

Cigna and the PXDX Controversy

A 2023 ProPublica investigation revealed that Cigna doctors rejected over 300,000 claims in a two-month period using a system called PXDX. Doctors spent an average of 1.2 seconds per claim — insufficient for meaningful review. A subsequent class-action lawsuit alleged that Cigna used the system to deny claims in bulk without individual physician review, as required by state law. In March 2025, a federal court allowed portions of the case to proceed. Cigna disputes that PXDX uses AI, calling it a “simple sorting technology.”

PM lesson: Even if your system isn’t technically ‘AI,’ automated decision-making at scale faces the same scrutiny. Document that human review is meaningful and timed, not ceremonial.

State Farm and Property Claims AI

In late 2025, homeowners filed a lawsuit against State Farm alleging the company used AI to systematically undervalue or deny property damage claims. The case illustrates that AI litigation is expanding beyond health insurance into property and casualty lines.

PM lesson: AI governance isn’t only for health insurers. Any automated system affecting coverage decisions is a potential target — regardless of the line of business. 

The Regulatory Landscape

Insurance is regulated primarily at the state level in the U.S., creating a patchwork of requirements. Coordination through the National Association of Insurance Commissioners (NAIC) is driving convergence — but PMs also need to track EU exposure and any jurisdiction-specific requirements.

NAIC Model Bulletin on AI (December 2023)

The NAIC adopted its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers in December 2023. By late 2025, 24 states plus Washington D.C. had adopted it. The Model Bulletin doesn’t create new law — it clarifies how existing insurance laws apply to AI. Core expectations include:

•       Written AI System Program: Establish governance with defined roles, risk controls, and documentation maintained for regulatory examination.

•       Consumer protection: AI-supported decisions must comply with unfair trade practice laws and must not result in unfair discrimination.

•       Third-party oversight: Insurers are responsible for AI systems they acquire from vendors. Due diligence, contractual protections, and audit rights are required.

•       Examination readiness: Regulators may request documentation on any AI system at any time. Be prepared to explain how systems work and demonstrate compliance.

State-Level Variations

Several states have gone further than the Model Bulletin:

•       Colorado AI Act (May 2024): Requires insurers to implement governance and testing procedures to prevent algorithmic discrimination, with explicit testing requirements beyond the Model Bulletin.

•       New York DFS Circular Letter 2024-7: Requires insurers to demonstrate AI systems don’t proxy for protected classes or generate disproportionate adverse effects. Insurers must maintain explanatory documentation and allow regulatory review of vendor tools.

•       California SB 1120 (effective January 2025): Regulates AI-enabled automated decision tools in health care claims processing.

EU AI Act — Annex III Classification

The EU AI Act explicitly classifies certain insurance AI applications as high-risk under Annex III, Article 5(c):

This classification triggers full Chapter III obligations for any insurer, reinsurer, or technology deployer operating in the EU: a risk management system maintained throughout the AI lifecycle, data governance ensuring training data is representative and free from errors, technical documentation demonstrating compliance, human oversight mechanisms with the ability to intervene, accuracy and robustness standards, and post-market monitoring.

The EU AI Act’s recitals further note that AI systems intended for health and life insurance risk assessment and pricing “can also have a significant impact on persons’ livelihood and if not duly designed, developed and used, can infringe their fundamental rights and can lead to serious consequences for people’s life and health, including financial exclusion and discrimination.” Property and casualty insurance AI may also qualify as high-risk if it involves profiling individuals.

Singapore FEAT Principles

Singapore’s Monetary Authority developed the FEAT principles (Fairness, Ethics, Accountability, Transparency) specifically for AI in financial services, including insurance. The Veritas framework provides a practical methodology for evaluating AI systems against these principles, with particular emphasis on fairness metrics for credit and insurance applications. 

Insurance-Specific AI Risks

Unfair Discrimination and Proxy Variables

Insurance has always involved risk classification — charging different prices based on actuarially valid risk factors. But AI can find correlations that effectively discriminate against protected classes, even without using protected characteristics directly.

Proxy discrimination occurs when a model uses variables — zip code, education level, consumer behaviour patterns — that correlate with race, gender, or disability, producing discriminatory outcomes without explicitly considering protected characteristics. NIST AI RMF MEASURE 2.11 requires that fairness and bias evaluations identify “input data features that may serve as proxies for demographic group membership.”

For underwriting and pricing AI, this means proxy variable analysis is not optional — it is a documented requirement under both NIST’s framework and the NAIC Model Bulletin’s unfair discrimination prohibition.

Lack of Explainability

When a customer asks “why was my claim denied?” or “why is my premium this amount?” — insurers need answers. Complex AI models may not produce clear explanations. Regulatory expectations are tightening: New York’s circular letter requires “explanatory documentation” for AI systems. The EU AI Act Article 13 requires that high-risk AI systems be designed to enable deployers to interpret outputs correctly.

Explainability isn’t just a customer service concern — it’s an examination risk. If your organization cannot explain a decision to a regulator, that is a compliance failure.

Data Quality and Historical Bias

Insurance AI systems are trained on historical data that may encode past discrimination. If historical claims data shows certain groups were denied more often — whether for legitimate risk reasons or not — an AI trained on that data may perpetuate the pattern. NIST AI RMF MEASURE 2.11 identifies systemic bias as a category that “can be present in AI datasets, the organizational norms, practices, and processes across the AI lifecycle, and the broader society that uses AI systems.”

NAIC surveys found that nearly one-third of health insurers still do not regularly test their models for bias — a gap regulators are actively working to close. 

Governance Framework for Insurance AI

Step 1: Risk Classification

Not all insurance AI carries the same risk. Governance intensity should scale with consumer impact and regulatory exposure.

Step 2: Required Documentation

Prepare for regulatory examination by maintaining a documentation set for every AI system. Regulators may request any of these items with little notice.

Step 3: Fairness Testing

For underwriting, claims, and pricing AI, implement systematic fairness testing before deployment and on an ongoing basis. NIST AI RMF MEASURE 2.11 provides the framework; the NAIC Model Bulletin and state regulations provide the compliance imperative.

•       Disparate impact analysis: Compare outcomes across demographic groups. Are certain groups denied more often, charged more, or experiencing different claim approval rates? Define acceptable thresholds before deployment.

•       Proxy variable analysis: Identify which input variables correlate with protected characteristics. Does your model use variables that effectively serve as proxies for race, gender, or disability?

•       Counterfactual testing: What happens if you change a single characteristic? Does the outcome change in ways that suggest discrimination?

•       Ongoing monitoring: NIST MANAGE 4.1 requires that monitoring effectiveness is evaluated continuously. Fairness metrics must be tracked in production, not just at initial validation.

Step 4: Human Oversight

Design meaningful human review into high-stakes decisions. The Cigna case demonstrates the legal and regulatory risk of oversight that is nominal rather than substantive.

•       Underwriting: Human review of adverse decisions, particularly for edge cases and appeals. Document who reviews, when, and with what authority to override.

•       Claims: Ensure claim denials receive actual physician or adjuster review. Track and audit review times. A 1.2-second average review is not a defensible oversight model.

•       Pricing: Human oversight of pricing model outputs with the ability to intervene on outliers. The EU AI Act Article 14 requires that deployers of high-risk AI systems are able to “correctly interpret the output” and “decide not to use the AI system in a particular situation.” 

Third-Party AI: Vendor Responsibility

Many insurers use AI systems from vendors — predictive models, claims processing tools, fraud detection services. The NAIC Model Bulletin is unambiguous: you are responsible for AI you deploy, regardless of who built it. NIST AI RMF MANAGE 3.1 echoes this: “AI risks and benefits from third-party resources are regularly monitored, and risk controls are applied and documented.”

Vendor Due Diligence

Contractual Protections

Ensure vendor contracts explicitly address:

•       Right to audit the AI system

•       Requirement to cooperate with regulatory examinations

•       Advance notification of material changes to the model

•       Liability allocation for discriminatory outcomes

•       Data handling, privacy, and security requirements

•       Performance guarantees and remedies for non-compliance 

PM Responsibilities by Phase

Planning

•       Identify which AI systems are subject to regulatory requirements — NAIC, state-specific, and EU AI Act for international operations

•       Classify risk levels for each system using the four-tier framework above

•       Include governance, documentation, fairness testing, and examination readiness in scope and budget

•       Engage compliance, legal, and actuarial stakeholders at project initiation — not after the model is built

Development and Procurement

•       Ensure fairness testing is part of validation criteria, with defined acceptable thresholds

•       Document model development decisions, data choices, and known limitations

•       For vendor solutions, conduct formal due diligence before contract execution

•       Establish explainability requirements — how will the model’s decisions be explained to customers and regulators?

•       Design human oversight mechanisms into the workflow, not as a post-deployment addition

Deployment

•       Verify governance documentation is complete and examination-ready before go-live

•       Confirm monitoring is operational, with appeal and overturn rate tracking in place

•       Train operators on oversight responsibilities, escalation triggers, and override authority

•       Establish customer feedback and appeal processes

Post-Deployment

•       Monitor for model drift and fairness metric deterioration — both trigger re-validation obligations

•       Review appeal and overturn rates on a defined schedule; escalate when thresholds are breached

•       Maintain documentation currency for regulatory examination readiness

•       Conduct periodic re-validation, including bias testing with current production data 

Key Questions for Insurance AI Projects

Use these questions to assess governance readiness before deployment.

Regulatory

•       Which states’ regulations apply to this system, and have we engaged compliance on each?

•       Is this system high-risk under the EU AI Act Annex III? If we operate in the EU, have Chapter III obligations been addressed?

•       Are we prepared for regulatory examination — today, not in six months?

Fairness

•       Have we tested for disparate impact across all relevant demographic groups?

•       Have we analyzed proxy discrimination risk in the model’s input features?

•       What is our process for addressing identified bias before and after deployment?

Transparency

•       Can we explain individual decisions to customers in plain language?

•       Can we explain how the system works to a regulator, including which variables drive outcomes?

•       Is our documentation examination-ready?

Oversight

•       Is human review substantive or ceremonial? Can we demonstrate review time and quality?

•       What triggers escalation from automated to human decision-making?

•       How do we handle appeals, and who has authority to override the AI?

Vendors

•       Have we conducted formal due diligence on every third-party AI system in scope?

•       Do our contracts include audit rights, change notification, and liability allocation?

•       Can our vendors support a regulatory examination if required? 

Right-Sizing for Your Situation

Insurance AI governance should scale with your regulatory exposure and the consumer impact of your systems. A marketing segmentation model needs substantially less governance than a claims adjudication system. AIPMO’s implementation playbooks provide practical guidance calibrated to your stage.

Become a member →

Framework References

•       EU AI Act (Official Journal, 12 July 2024) — Annex III, Point 5(c) (insurance risk assessment and pricing as high-risk); Article 13 (transparency and provision of information); Article 14 (human oversight); Chapter III Section 2 (requirements for high-risk AI systems); Recital 58 (insurance-specific risk rationale)

•       NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1) — MEASURE 2.11 (fairness and bias evaluation and documentation); MEASURE 2.9 (model explainability and interpretability); MANAGE 3.1 (third-party AI risk monitoring and controls); MANAGE 4.1 (monitoring effectiveness)

•       NIST AI RMF Playbook — MEASURE 2.11 suggested actions (disparate impact analysis, proxy variable analysis, counterfactual testing, demographic parity metrics)

•       IAPP / HCLTech Global AI Governance Law & Policy Series 2025 — U.S. regulatory landscape including EEOC AI and Algorithmic Fairness Initiative; EEOC v. iTutorGroup settlement; Mobley v. Workday vendor liability implications

•       NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers (December 2023) — governance program requirements, consumer protection obligations, third-party AI accountability

•       AIGP Body of Knowledge v1.0.0 — Domain III (non-discrimination laws applicable to insurance AI; unfair and deceptive practices; requirements for human supervision of algorithmic systems)

•       PMI Guide to Leading and Managing AI Projects (CPMAI 2025) — Phase I (stakeholder impact assessments and fairness requirements); Phase V (fairness testing across demographic groups); Phase VI (ongoing bias monitoring in production)

This article is part of AIPMO’s Sector series. See also: AI Risk Classification | Third-Party AI and Vendor Management | Human Oversight in AI Systems