Your Health System Has Shadow AI. Who's Actually Governing the Agents?

I once watched a Stripe dashboard hit a million dollars in a single day. I refreshed it like a lunatic, grinning, feeling like I'd finally cracked something that everyone else had missed. The product was bad. Not catastrophically bad, just quietly, stubbornly bad in ways that a million dollars of revenue made very easy to ignore.

Scale doesn't fix problems. It hides them, and then, quietly, while you're busy hitting refresh, it multiplies them.

I thought about that dashboard this week while reading a new Imprivata survey on agentic AI in healthcare. More than 85% of AI leaders said they're confident they can see and control what their AI agents do. And 72% admitted AI tools get deployed without IT approval at least occasionally.

They can't both be true. One of them is the dashboard.

What is shadow AI in a health system?

Shadow AI is any AI tool running inside your organization that nobody with authority approved. According to Healthcare Dive's coverage of the Imprivata report, 72% of healthcare AI leaders say it happens at least occasionally, and a separate Wolters Kluwer survey found 40% of medical staff know colleagues using unauthorized AI tools.

Nearly one in five admitted doing it themselves.

That's not a rogue employee problem. That's a demand signal. People are drowning, the sanctioned tools are slow or missing, and a free chatbot is one browser tab away. (I'd probably do it too, honestly.)

Why are AI agents riskier than the software we already govern?

Because agents act. Traditional software waits for a human to click something, while agents move across systems and execute on someone's behalf. Imprivata's Dr. Sean Kelly laid out the danger plainly: an agent with excessive permissions could expose patient data, write the wrong thing into a record, or change a dosage under a clinician's authority.

And it happens at machine speed.

A tired nurse makes one error and someone usually catches it by shift change. An agent can make the same error four thousand times before lunch. Adoption isn't slowing down either. More than a quarter of organizations have already implemented agentic AI, 44% are piloting it, and another 21% plan to within a year.

What does good AI agent governance actually look like?

It looks a lot like how you already govern people. Every agent needs its own identity, a defined set of permissions, clear boundaries on what it can touch, human review that scales with clinical risk, and an audit trail you can actually read. As Kelly put it, "Oversight must match the level of clinical risk."

None of that's exotic.

It isn't new advice either. ECRI ranked insufficient AI governance the number two patient safety concern of 2025, citing a survey where only 16% of hospitals had system wide AI governance policies. The gap was flagged. Then the agents showed up faster than the committees did. Smaller organizations look the same, by the way: 58% of practices in Weave's 2026 survey have no formal AI governance structure.

Can you govern an AI vendor you can't inspect?

Not really. You can govern the contract, the BAA, and the access list, but if the model, the prompts, and the escalation logic live inside someone else's black box, your audit trail ends exactly where their API begins.

That's why we built HANA fully open source and self hosted, with no OpenAI dependency.

Your security team can read the code. Patient conversations stay inside your perimeter. Permissions and boundaries become configuration you own, not a promise in a slide deck. If you want to see what that looks like in practice, our technical documentation covers deployment, integration, and where every piece of data lives.

How do you keep a patient facing AI agent safe at scale?

Keep its job narrow and its exits obvious. A safe patient facing agent does one clinical workflow well, knows exactly which answers are out of bounds, and hands off to a human the moment something sounds wrong. It never improvises clinical judgment.

I once had a breakdown in a meeting. Crying, couldn't stop, brain completely fried. For months I'd been running past my limits and nobody, including me, was watching the gauges. The body keeps score.

Agents don't cry in meetings. That's the scary part. They fail quietly unless you've built the gauges in.

That design discipline is how we've run over 1M patient interactions across 5 countries and 3 languages with zero critical adverse events. Our case studies show what the escalation paths look like in live deployments.

Does governance slow down patient engagement?

It shouldn't, and if it does, you'll get more shadow AI. Staff route around governance when the approved option is worse than the unapproved one. So the sanctioned tool has to actually win.

That's the part committees tend to forget. A perfectly governed tool nobody uses is safe and useless.

Our weekly patient engagement sits at 85%, against a 15 to 20% industry baseline, and we treat that number as a governance metric, not a marketing one. If patients and staff aren't using the approved path, they're using some other path you can't see. Our research page breaks down the engagement data.

Key Takeaways

Shadow AI in healthcare isn't a discipline problem. It's a signal that staff need better tools than the ones they've been given. Agentic AI raises the stakes because agents act on their own at machine speed, so governance has to cover identity, permissions, boundaries, human review scaled to risk, and an audit trail someone can read. You can't fully govern what you can't inspect, which is the strongest argument for open source, self hosted infrastructure in patient facing AI. And governance only works if the approved tool is good enough that nobody wants to route around it.

FAQ

What's the difference between agentic AI and a chatbot?

A chatbot answers. An agent acts: it can read systems, make multistep decisions, and execute tasks like scheduling or documentation on someone's behalf. That autonomy is why agents need their own permissions and audit trails.

Who should own AI agent governance in a health system?

Not one department alone. ECRI recommends a multidisciplinary committee spanning clinical leadership, IT, security, patient safety, and risk. What matters most is one clear owner per agent, so accountability never falls between teams.

Is open source AI safe enough for patient conversations?

Open source doesn't make a system safe by itself, but it makes safety verifiable. Your team can audit the code, host it inside your own perimeter, and control exactly what the agent can access. With a closed vendor, you're trusting a claim.

If you're working out how to bring patient facing agents in without adding to the shadow pile, book 30 minutes with me.