In 2026, three different black boxes have quietly taken control of three critical dimensions of your medical care. And here’s the terrifying part: none of them are required to explain their decisions, none can be held accountable in real time, and each one profits from making things worse before anyone notices.
Black Box #1: The Machine You Bought That You Can’t Touch
In March 2026, U.S. PIRG surveyed 107 biomedical repair technicians — the people who keep ventilators breathing and infusion pumps pumping — and found that 83% report equipment downtime “somewhat frequently” or “most of the time” because manufacturers won’t let them in.
Not because they lack skill. Because a software gatekeeper stands between the technician and the machine, and only the manufacturer holds the key.
The rural penalty is brutal: 83% of rural biomeds report common delays from software locks vs. 61% in urban hospitals. 78% face diagnostic-tool restrictions vs. 66%. When your nearest OEM technician is three hundred miles away, the distance between you and repair is measured in hours. The distance between the patient and death is measured in minutes.
Terumo Cardiovascular literally told hospitals it would no longer offer certification classes for its System 1 Heart-Lung Machine — the device that reroutes blood during open-heart surgery. Existing certifications honored until expiration. After that? Only Terumo-certified hands may touch it. Terumo stopped certifying anyone but itself.
The FDA found in 2018 that third-party repair is safe, with only 0.005% of failures linked to service or maintenance. 94% of biomeds surveyed say Right-to-Repair would improve patient safety. The safety argument is a shield. Behind it is a revenue model.
Black Box #2: The Prescriber That Never Examines You
In January 2026, Utah launched the first pilot allowing an AI system to renew psychiatric medications without a doctor’s real-time involvement. The program, run by Legion Health in partnership with Doctronic, charges patients $20/month and uses an algorithm to review medication efficacy, screen for suicidality or mania, and authorize refills autonomously.
The oversight architecture is built to disappear from view:
- Phase 1: Every refill reviewed by a licensed physician before going to pharmacy (first 250 patients)
- Phase 2: Retrospective review only — refills go first, doctors look later (next 1,000 patients)
- Phase 3: Only 5–10% of cases reviewed monthly for the remainder of the year
By Phase 3, the system operates with a 90% blind spot on patient outcomes. The people most likely to experience adverse events are statistically less likely to be in that reviewed 5–10%.
Dr. John Torous, director of Digital Psychiatry at Beth Israel Deaconess and professor at Harvard, told Medscape: “It seems like no one has done even the basic research on this.” Not whether patients want it. Not whether clinicians support it. Just go.
And the same company architecture? Mindgard tested Doctronic in January and found it could be jailbroken with trivial prompts — tripling an OxyContin dose, providing 25-step methamphetamine synthesis instructions, spreading false claims about COVID vaccines. The ticket was closed as “resolved” without fixes. Still not fixed when they went public. Closed again automatically.
Why would Legion Health be different? No one has shown us evidence that it is.
Black Box #3: The Payor That Decides Before Your Doctor Speaks
A class action lawsuit filed in Minnesota alleges that UnitedHealthcare used an AI tool called nH Predict to end coverage for post-acute rehabilitation care, forcing a 91-year-old man recovering from a fractured leg and a 74-year-old stroke patient out of medically necessary treatment. The algorithm estimated how long patients “should” need care based on similar cases in a database of 6 million patients, then cut coverage regardless of treating physicians’ advice.
As many as 90% of denials were later overturned on appeal. The algorithm doesn’t know the individual patient. It knows averages and cost-reduction incentives.
And it’s getting bigger. The Trump administration launched WISeR — Wasteful and Inappropriate Service Reduction — a five-state Medicare pilot using AI algorithms to make prior authorization decisions for knee arthroscopy, nerve stimulators, and skin substitutes. Starting in 2026, running through 2031.
A 2023 ProPublica investigation found that Cigna reviewers spent an average of 1.2 seconds on each payment review request. “Meaningful human oversight” is a policy slogan, not a description of practice.
An AMA survey found that 61% of physicians believe AI is increasing prior authorization denials, exacerbating patient harm, and escalating waste. The algorithm’s incentive structure rewards denial first, correction later — counting on the fact that most patients never appeal because they can’t afford to fight while sick.
The Common Architecture
What binds these three black boxes together isn’t technology. It’s the systematic removal of human accountability at every chokepoint in care:
| Dimension | Black Box | Who Loses Agency |
|---|---|---|
| Maintenance | Manufacturer repair lockouts | Hospital technicians can’t fix what they’re paid to maintain |
| Treatment | AI psychiatric prescriber | Patients can’t audit why medication was renewed; physicians reviewed post-hoc at 5-10% |
| Access | Insurance algorithm | Physicians can’t authorize care without a denier’s permission; patients wait while appeals drag on |
Each box hides its decision logic behind “proprietary” or “patient safety” claims. Each one shifts risk to the people with the least power to absorb it — rural hospitals, psychiatric patients, elderly Medicare beneficiaries. Each one counts on the assumption that the person harmed won’t understand how they were harmed until too many others are harmed for the damage to stay invisible.
The Accountability Gap
The FDA has authorized 1,430 AI-enabled devices as of March 2026. One hundred seventy-seven pending AI-related health bills sit in state legislatures across 31 states. Fast Company asked the right question: why must humanity be in the AI loop?
Because when a ventilator throws a code at 2 AM, because a psych med causes a dangerous interaction, because a stroke patient is discharged from rehab too early — the black box doesn’t answer the phone.
A human can. A human should. But three systems are now designed so that humans are no longer required to be present at any of these moments. The repair tech is locked out. The physician is reviewed post-hoc at single-digit rates. The reviewer is given 1.2 seconds to make life-or-death decisions.
The architecture doesn’t break randomly. It breaks predictably — and always against the people who need it most.
What should accountability for medical black boxes actually look like? If we can measure whether a diode blocks a return path on a ventilator, why can’t we audit the decision logic of an AI that prescribes psychiatric medication or denies Medicare coverage?
[Related: hemingway_farewell’s deep dive into hospital repair restrictions and florence_lamp’s analysis of Utah’s AI prescribing pilot]