Avoiding Missteps: 3 Keys to Governing, Scaling and Deploying AI Responsibly

Ask any health system leader what keeps them up at night, and AI governance is increasingly near the top of the list. Not because they're afraid of the technology but because the stakes of getting it wrong are high, and the playbook is still being written.

At the AHA Leadership Summit in Denver last week, three healthcare leaders and an AI platform executive offered an experience-driven look at what responsible AI adoption requires: how to build governance that holds; how to think about AI autonomy without losing sight of the opportunity; and how to sequence adoption so the right tools get deployed for the right reasons. Here's what they shared.

1 | Build governance that has real teeth.

Most health systems have some form of AI governance, but many could still benefit from strengthening it to more effectively block bad decisions.

When CommonSpirit Health was renegotiating its ambient AI scribe contract, the vendor couldn't satisfy the system's requirements around patient data use. The response was swift. "We said, 'You know what? You're out,'" said Daniel Barchi, CIO of the 150-hospital system. That call came from CommonSpirit's 25-member Enterprise Data and Governance Committee, a cross-functional body spanning technology, clinical, ethics, mission and finance that has rejected 17 proposed use cases to date.

At Intermountain Health, a Salt Lake City-based nonprofit system with 33 hospitals and more than 400 clinics, CEO Rob Allen took a different structural approach. Rather than creating a standalone AI committee, he embedded AI accountability directly into every existing board committee's charter. PULSE teams — cross-functional groups responsible for prioritization, uncovering risks, legal and compliance, strategic alignment and execution — evaluate, approve and execute AI within their domain with a target turnaround from idea to go/no-go of 30 days.

WellSpan Health, a not-for-profit health system with nine hospitals primarily serving rural and small communities, took a partnership-first approach to governance, co-developing solutions with partners using clear go/no-go criteria. What started five years ago with a single AI imaging solution supporting radiologists has grown to 21 deployed solutions across the system, said COO Kasey Paulus.

Saad Bashir of Amigo AI, a clinical platform that helps health systems build, train and deploy AI agents, argued that most governance frameworks are missing critical dimensions. Effective governance, he said, must address:

• Technical (which models, what cost, what latency)
• Behavioral (how an agent communicates with patients, what tone it uses, how it adapts across demographics)
• Clinical (whether outputs stay within defined guardrails).

Miss any one of them, he warned, and "it's like the McDonald's of governance. It tastes like food. It smells delicious. But when you look at it, you know something is missing — maybe the beef."

Takeaway

Governance that can't say no isn't governance. Build cross-functional oversight with real authority and make sure it meets frequently enough to keep pace with how fast AI is moving.

2 | Autonomy is a dial, not a switch. Move it carefully.

As AI tools become more capable, the question of how much autonomy to grant them is one of the most consequential decisions health system leaders face. The panelists were aligned: Human oversight remains essential, but how you define and structure it matters enormously.

WellSpan's agentic AI assistant ANNA handles more than 140,000 patient calls per month in multiple languages with no clinician in the room. And yet ANNA's scope is tightly bounded, using patient outreach to close care gaps and safety monitoring only. Every interaction is reviewed by a live human before the system advances toward greater autonomy. For Paulus, the principle is straightforward: "We have a live human reviewing all of these cases upfront. We solve for safety in AI the same way we solve for safety in healthcare."

At CommonSpirit, a clinician will always sit between any AI tool and any patient for autonomous diagnosis or action. "There is no special rule that AI gets to play by," Barchi said.

But he also pushed leaders to pair caution with ambition. CommonSpirit's governance dashboard tracks 260 active AI tools — and deliberately posts one additional data point alongside those metrics: 120. That's the number of people who died in North American hospital parking lots and garages last year. The message is intentional: Risk is everywhere in healthcare, and treating AI as uniquely dangerous while ignoring its potential is its own kind of failure.

Allen shared a cautionary example from a recent Intermountain deployment: A clinical summarization tool encountered a nurse's note about housekeeping turning over a room and reported that the care team had "cleaned the patient."

"That was really not what was in the record," Allen said. "But AI summarized it that way. And it's those things we have to watch for."

Bashir offered four questions to ask any AI partner before expanding a tool's autonomy:

• Was it built specifically for your organization or configured off the shelf?
• Was validation done on your patient demographics or a shared model?
• Can you see the reasoning behind outputs?
• In live use, is the agent improving based on your patients, or on a general shared model?

Takeaway

Move the autonomy dial based on evidence, not confidence. The subtle errors — not the obvious ones — are what will get you.

3 | Sequence AI adoption around mission, not vendor hype.

With hundreds of potential use cases competing for attention, how health systems decide where to start may matter as much as what they ultimately deploy.

For Intermountain, Allen said, the first filter is always mission: Does this advance what we're here to do? "If the answer is no, it's off the table immediately." From there, PULSE teams carry prioritization within their respective domains, so clinical teams aren't competing against revenue cycle for the same resources. Intermountain's two system-wide AI priorities are advancing proactive care and simplification.

"Einstein said the key to universality is simplification," he said. "If these tools don't simplify for us, we shouldn't be pursuing them."

Barchi pointed to two CommonSpirit use cases as proof of what mission-driven sequencing looks like in practice. The first: an AI screening tool embedded in multiple EHRs that flags candidates for lung and colon cancer screening and surfaces a prompt at the point of care. The second — and the one he's most proud of — is an internally developed tool deployed in CommonSpirit's Arizona EDs that identifies potential human trafficking victims by recognizing subtle clinical patterns no individual clinician could reliably catch. "We're catching them and really changing the lives of some young people who have been abused. I wish we could get to scale more quickly."

For community health systems, Paulus offered a practical on-ramp: Start with solutions that can generate quick, visible wins. Build the governance muscle through low-risk use cases before tackling the complex ones. Bashir echoed that directly: "Don't get excited from a very nice vendor presentation and start implementing that use case. You need to grow that governance muscle, and it only happens with some of those low-hanging use cases."

Takeaway

The health systems making the most progress aren't chasing the flashiest applications. They're building the organizational infrastructure — governance, sequencing, finance validation — that lets them scale responsibly when the moment is right.

The Bottom Line

AI in healthcare is no longer a future-state conversation. Health systems are deploying hundreds of tools, handling millions of patient interactions and making real-time decisions that affect care. The leaders who will come out ahead aren't necessarily the fastest movers — they're the ones building governance with real authority, defining autonomy with precision and sequencing adoption around mission.