Here’s something that might surprise you: most of the AI transforming healthcare today isn’t actually regulated by the FDA. While everyone’s focused on whether AI can accurately read X-rays or diagnose cancer, there’s a massive ecosystem of health AI tools operating in what’s essentially a regulatory no-man’s land.
A new report from the Bipartisan Policy Center dug into this mess, and what they found is both fascinating and a little concerning. We’re talking about AI that’s already embedded in hospitals, clinics, and our phones—making decisions that affect our health—but flying under the radar of traditional medical device oversight.
The AI That’s Already Here (And Not Going Anywhere)
Let’s start with what we’re actually talking about. When most people think “medical AI,” they picture robots performing surgery or AI diagnosing diseases from medical scans. But that’s just the tip of the iceberg.
Take administrative AI. Right now, there are algorithms deciding whether your insurance will approve that MRI your doctor ordered. AI systems are flagging potentially fraudulent medical bills, figuring out how many nurses a hospital needs next Tuesday, and managing appointment schedules. None of this feels particularly “medical,” but it’s all happening in healthcare, and it all affects patients.
Then there are the clinical support tools living inside electronic health records. These systems are constantly analyzing patient data, popping up reminders like “Hey, this patient is overdue for a colonoscopy” or “This person’s blood pressure pattern looks concerning.” They’re not making diagnoses, but they’re definitely influencing care decisions.
And finally, there’s all the consumer stuff we use every day. Your Apple Watch tracking your heart rate, that meditation app helping you sleep, the fitness tracker counting your steps—technically, it’s all health AI, and it’s all generating data about your wellbeing.
Where the 21st Century Cures Act Drew the Line (Sort Of)
Back in 2016, Congress tried to sort this out with the 21st Century Cures Act. The idea was simple: create clear rules about when health software needs FDA oversight and when it doesn’t.
They came up with four criteria for AI tools to avoid FDA regulation. The tool can’t analyze medical images or body signals (so no reading X-rays or interpreting EKGs). It has to work only with information already in the patient’s chart. It can only support medical decisions, not make them. And doctors have to be able to understand and review what the AI is recommending.
Sounds reasonable, right? Here’s the catch: miss any one of these criteria, and boom—your AI tool suddenly becomes a medical device that needs FDA approval. It’s like a regulatory cliff. You’re either completely exempt or you’re in full medical device territory. There’s no middle ground, which has created some pretty confusing situations for developers.
The Alphabet Soup of Agencies Trying to Keep Up
So who’s watching all this non-FDA AI? Well, that’s where things get interesting. It turns out there are a bunch of different agencies, each with their own piece of the puzzle.
The Office of the National Coordinator for Health IT requires some transparency, but only if your AI is built into a certified electronic health record system by the EHR company itself. Third-party apps or homegrown hospital AI? They’re mostly on their own.
The Office for Civil Rights jumps in whenever patient data is involved, which is pretty much always. They’re the HIPAA enforcers, and they’re also supposed to make sure AI isn’t discriminating against certain groups of patients.
The Federal Trade Commission goes after companies making bogus claims about what their AI can do. You know, the “This app can cure your diabetes!” type of stuff. They also make non-medical health apps report data breaches.
The Centers for Medicare & Medicaid Services doesn’t directly regulate AI, but they have huge influence through what they’ll pay for and what requirements they put on hospitals and doctors.
And then there are the states, which are increasingly doing their own thing. Colorado passed a comprehensive AI risk law. Illinois and Utah have their own disclosure rules. Some states are creating “regulatory sandboxes” where companies can test AI tools under relaxed rules to see what works.
Why This Patchwork Approach Is Both Great and Terrible
On one hand, this fragmented system has some advantages. It’s flexible. It allows innovation to happen without getting bogged down in bureaucracy. Different types of AI can be overseen by agencies that actually understand their specific risks and benefits.
But it’s also kind of a nightmare. Companies building AI tools often can’t figure out which rules apply to them. Healthcare providers are confused about what they’re allowed to use. And patients? Well, they’re mostly just hoping someone, somewhere, is making sure all this AI is safe and effective.
The biggest concern is the gaps. With so many different agencies involved, it’s entirely possible for problematic AI to slip through the cracks. Maybe it doesn’t quite meet the threshold for FDA oversight, but it’s also not clearly covered by any other agency. That’s a recipe for trouble.
What Happens Next?
The folks at the Bipartisan Policy Center think we need clearer frameworks and better coordination between all these different regulators. That makes sense, but it’s easier said than done. Getting federal agencies, state governments, and industry groups to agree on anything is like herding cats.
The stakes are real, though. Healthcare AI has enormous potential to make care better, cheaper, and more accessible. But realizing that potential requires getting the regulatory balance right. Too little oversight, and we risk patient safety and privacy. Too much, and we might stifle innovation that could save lives.
As Jonathan Burks from BPC put it, “The health care AI revolution is well underway, transforming how care is delivered and raising new questions about regulation.” He’s right. The AI is already here, already making decisions, already changing healthcare. The question isn’t whether we need to regulate it—it’s how to do it smart.
The reality is that we’re figuring this out as we go. Healthcare AI is evolving faster than our ability to regulate it, which means we’re constantly playing catch-up. The key is making sure that as we develop new rules and frameworks, we’re protecting patients without killing the innovation that could dramatically improve healthcare for everyone.
It’s a tricky balance, but it’s one we have to get right. Because ready or not, the future of healthcare is already here—and it’s powered by AI.
Sources
- Pennic, Fred. “Oversight Beyond the FDA: New Report Untangles the Complex Regulation of Health AI Tools.” HIT Consultant, June 23, 2025.
- Bipartisan Policy Center. Issue Brief: Health AI Tools Regulation Beyond FDA Jurisdiction, 2025.
- 21st Century Cures Act, Pub. L. No. 114-255 (2016).
- U.S. Food and Drug Administration. Software as a Medical Device (SaMD) guidance documents.
- Office of the National Coordinator for Health Information Technology (ONC). Health IT Certification Program requirements.
- Department of Health and Human Services, Office for Civil Rights. HIPAA Privacy and Security Rules.
- Federal Trade Commission. Health Breach Notification Rule and consumer protection enforcement actions.
- Centers for Medicare & Medicaid Services. Conditions of Participation and reimbursement policies.
- State AI legislation: Colorado AI risk assessment law, Illinois AI disclosure requirements, Utah consumer protection AI laws.
Discover more from Doctor Trusted
Subscribe to get the latest posts sent to your email.
