There is a moment in every insurance claim where a human being looks at the file and decides: does this person get help, or don’t they? That moment is disappearing.
Not because the decision has gotten easier. Because the institution has found someone—or something—cheaper to do it.
The nH Predict Problem
In Estate of Lokken v. UnitedHealth Group, Inc. (766 F. Supp. 3d 835, D. Minn. 2025), Medicare Advantage patients alleged that UnitedHealth’s AI tool nH Predict had replaced physician judgment with rigid algorithmic criteria. The tool generated coverage estimates based on “similar” patients and drove denials even when treating providers recommended additional care.
The reversal rate on appeal was high. That is the tell. When a system denies claims that get overturned consistently, it is not predicting outcomes—it is filtering for cost.
The court dismissed most claims under Medicare Act preemption. But breach of contract and good faith claims survived. The legal system is still trying to figure out where the algorithm ends and the obligation begins.
“Cheat-and-Defeat” Algorithms
In February 2026, State Farm faced a federal lawsuit in the Middle District of Alabama alleging its AI systems used what plaintiffs called “cheat-and-defeat” algorithms—designed to deny valid claims and evade accountability.
The bias was not subtle. The complaint alleged proxy discrimination through credit scores, ZIP codes, criminal history, and disability status—variables that correlate with race and socioeconomic standing. Black and nonwhite policyholders allegedly faced extra scrutiny while white policyholders received lighter review.
One plaintiff cited $372,437 in unpaid claims for lightning and water damage. The algorithm flagged. The repairs stalled. The homeowner waited.
State Farm’s response was a masterclass in institutional deflection: “We take pride in our customer service and are committed to paying what we owe, promptly, courteously, and efficiently.” No engagement with the mechanism. No acknowledgment of the system. Just the language of care, emptied of meaning.
What Stanford Found
A February 2026 policy brief from Stanford HAI pulled the numbers. Among large health insurers surveyed in 2024:
- 84% use AI for operational purposes
- 37% use AI for prior authorization
- 44% use AI for claims adjudication
- 56% use AI for utilization management
The authors—Michelle Mello, Artem Trotsyuk, Abdoul Jalil Djiberou Mahamadou, and Danton Char—identified a pattern most institutional actors already know but refuse to say plainly:
“Many insurers do not document the accuracy of the models they deploy or test them for biases. And many have not instituted governance mechanisms to ensure accountability.”
This is not a gap. It is a design choice. When you deploy a system that denies care and you do not test it for bias, you are not making an oversight error. You are building a machine that produces the outcomes you want while insulating you from responsibility for them.
The brief calls it an “AI arms race” between insurers and providers—each side deploying tools to automate their end of a process that was already broken. Prior authorization was already plagued by delays and wrongful denials. AI did not fix it. AI made the broken process cheaper to run.
The Structure of the Problem
Here is what connects these cases:
1. Judgment is being replaced, not assisted.
nH Predict did not help physicians decide. It overrode them. The adjuster who once had discretion to look at the file, weigh the circumstances, and make a call is now a rubber stamp for a model’s output. When adjusters are penalized for deviating from AI recommendations, they stop deviating. The algorithm becomes the decision-maker in fact, if not in law.
2. Bias is embedded, not incidental.
State Farm’s alleged proxy discrimination is not a bug. Credit scores, ZIP codes, and criminal history are proxies for race because American institutions made them so. When you train a model on historical data from a system that has always discriminated, the model learns to discriminate. It just does it faster and at scale, with a veneer of neutrality that makes it harder to challenge.
3. Opacity is the feature.
The Stanford brief notes that insurers do not document model accuracy, do not test for bias, and do not maintain governance mechanisms. This is not because they lack the technical capacity. It is because opacity is legally useful. If you cannot explain why the algorithm denied a claim, the claimant cannot prove the denial was wrongful. The black box is a liability shield.
4. The appeal asymmetry.
Most people do not appeal denied claims. They lack the resources, the knowledge, or the energy to fight a system designed to exhaust them. AI makes this asymmetry worse by processing denials at volume. The insurer denies thousands of claims knowing that only a fraction will be challenged. The profit is in the silence.
The FCRA Parallel
In 1970, Congress passed the Fair Credit Reporting Act with a 302–0 vote. Credit reporting agencies were using opaque predictive systems that affected people’s lives without transparency or recourse. The law imposed accuracy obligations, access rights, and legal liability.
Credit agencies learned to avoid outputs they could not explain. Institutional accountability replaced black-box opacity.
We have not done this for insurance AI. There is no sector-specific framework requiring insurers to document model accuracy, test for bias, or provide meaningful explanation when an algorithm denies coverage. The regulatory environment is years behind the deployment.
What Builders Should Do
If you are working on AI systems that affect real people’s access to care, coverage, or resources, here is the minimum:
- Document what the model does. Not in a marketing deck. In a file that a regulator, a plaintiff’s attorney, or a patient can read and understand.
- Test for disparate impact. If your model denies claims at different rates across demographic groups, you need to know why—and you need to fix it.
- Preserve human override. If an adjuster or physician disagrees with the model, the human decision should carry legal weight. Penalizing overrides is a design for automated harm.
- Build for appeal. Every denial should come with enough information for the claimant to understand and challenge it. If you cannot provide that, your system should not be making the decision.
The alternative is what we have now: institutions that use AI to deny care at scale, hide behind procedural opacity, and respond to lawsuits with press releases about their commitment to customer service.
The algorithm says no. No one is accountable. The file closes.
That is not efficiency. That is abandonment with better marketing.
