Nobody Wants To Put Their Name On The Algorithm

In 2016 I watched a Stripe dashboard tick past a million dollars in a single day. My DTC company. My product. I remember the specific feeling, which was not joy, it was vertigo, because I already knew the product wasn't good. Scale doesn't fix a bad foundation. Scale just makes the crack louder.

We went to $12M and then it blew up. Not because demand died. Because everything underneath the demand was held together with duct tape and optimism, and at a certain volume duct tape stops being a strategy.

I think about that every time a health system tells me their AI pilot went great.

Why do healthcare AI pilots succeed and then die in production?

Because pilots are allowed to cheat. A MedCity News piece on healthcare AI architecture put it better than I could: pilots tolerate exactly the shortcuts that production can't. A one off data extract for the vendor. A nightly refresh because nobody wanted to fight the EHR vendor over API quotas. A custom integration someone built on a weekend.

The clinical champion gets a clean demo. The budget owner gets a believable ROI number. Someone signs off.

Eighteen months later there are four of these running side by side, the integration team is on call most weeks, and two AI tools are quietly disagreeing about the same patient because they're reading different snapshots of the same chart. Neither one is wrong. That's the part that should scare you.

What is the actual failure, if it isn't the model?

Ownership. Somebody has to be willing to put their name on the decision the model makes, and in most organizations that person doesn't exist yet.

Ask the four questions that kill deployments and watch the room go quiet. Where does the model run. Who owns it on call at 2am. Which compliance regime governs its outputs. What audit trail exists when it's wrong.

If those weren't answered in week one, they show up in month fourteen as objections nobody has the authority to resolve. Then the pilot ends. Quietly, the way these things always end, with a calendar invite that stops recurring.

Should governance live in the tool or underneath it?

Underneath. Always underneath.

When governance is retrofitted into each application, every vendor's model update breaks something, and you're reconstructing decisions from memory when an auditor asks about a determination from eighteen months ago. Model versioning, lineage, audit logging, human review checkpoints, these belong in the shared data and integration layer, not bolted onto whatever each vendor shipped last quarter.

This is the boring work. It does not make a good board slide. That's precisely why it keeps getting deferred, and why the systems doing it now will spend the back half of the decade compounding while everyone else does cleanup.

We built HANA fully open source and self hosted for exactly this reason. Not ideology. If you can't read the thing, version the thing, and audit the thing, you don't own it. You have a vendor relationship and a liability exposure.

Why start with operations instead of clinical AI?

Because the operational layer is load bearing and the clinical layer isn't.

Farid Fadaie calls this Operations-First AI, and he's right. A brilliant diagnostic model is worth close to nothing in a practice that can't book patients, can't reach them in their language, and loses a third of its calls to voicemail. You've built a beautiful room on a cracked slab.

Operational AI is repetitive, bounded, verifiable, and deployable now. Access, intake, scheduling, follow-up. High volume, measurable, and the part every patient actually experiences. Clinical AI compounds on top of it later, and it's worth far more once the foundation can act.

My daughter is ten. I tell her, and I mean it, that I work for her and not the other way around. Same logic applies here. The infrastructure works for the care, or you've inverted something important.

How do you know a system will survive production?

You watch it on an average day, not a good one.

A demo is the noise free call. Production is the interrupted sentence, the mid conversation language switch, the wrong number given and corrected, the patient who hangs up and calls back angry. MIT Technology Review argued in September that healthcare AI's next test is integration rather than model capability, and that's the same claim from a different angle. Capability is table stakes now. Orchestration and governance are the differentiators.

We've run 1M+ patient interactions across five countries and three languages with zero critical adverse events, and the outcomes are published because unpublished outcomes are marketing. The deployments themselves are messier and more useful than any benchmark, which is also true of the story of how we got here.

Key Takeaways

The AI was never the hard part. The hard part is a governed data layer every tool reads from, an accountability structure with a real human name attached, and the discipline to build operations before clinical.

Pilots that cheat produce production systems that collapse. If your vendor's integration bypasses your data layer, you're not buying capability, you're buying debt, and the interest rate goes up with every tool you add. Ask the 2am question before you sign, not after.

I learned the scale lesson at $12M and it cost me the company. In healthcare the bill lands somewhere worse than a balance sheet. Build the slab first.

FAQ

What's the single biggest reason healthcare AI fails to scale? Missing architecture and governance, not model quality. Most stalled deployments trace back to unanswered questions about ownership, runtime environment, compliance authority, and audit trails, all of which should be resolved before a model is built.

Should we build the data layer before buying more AI tools? Yes. A curated, governed, FHIR native layer that every tool reads from prevents each vendor from building its own bypass to the EHR. Retrofitting that layer later costs a multiple of building it first.

Is open source meaningfully different for compliance? It changes what you can prove. Self hosted, open source systems let you version, inspect, and audit the exact behaviour that produced an output, which is the thing regulators and plaintiffs' attorneys eventually ask about.

If you're staring at three AI tools and no shared foundation, that conversation is worth an hour. Book a discovery call and bring the architecture diagram, not the pitch deck.