AI Agents in Healthcare: What You Can Automate and What Compliance Requires

AI agents in healthcare can safely automate administrative and revenue-cycle preparation today: eligibility checks, prior authorization packets, coding suggestions, denial appeal drafts, referral intake, scheduling, and draft patient messages. A licensed or accountable person still signs anything clinical, anything sent to a payer, and anything adverse to a patient.

Compliance comes from five sources: HIPAA, FDA’s line between decision support and a device, federal transparency and nondiscrimination rules, CMS payer rules, and state AI laws. Several moved in 2026. Below: a use-case table, each rule’s status as of October 15, 2026, and a pass/fail checklist. I am not a lawyer; this is an operator’s map.

What can AI agents in healthcare automate safely today?

The risk of a healthcare agent depends less on the model than on where its output lands. Sort tasks into three tiers by landing zone.

Tier 1: internal and reversible. The output lands in a staff work queue. A wrong eligibility summary costs rework, and nobody outside sees it.

Tier 2: leaves the building after sign-off. The output goes to a payer, a patient, or the medical record, after a named person approves it. Prior auth submissions, appeals, coded claims, portal replies, and scribe notes live here.

Tier 3: clinical judgment. The output shapes a diagnosis, a treatment, or a coverage denial. The agent may surface evidence. A clinician decides, and the software may be a regulated device.

Decision rule: if a wrong output can reach a patient, a payer, or the chart before a person sees it, it is Tier 2 at minimum.

The 2025 CAQH Index, released February 19, 2026 from data on more than 600 provider organizations and health plans covering 63% of insured lives, found that more than 50% of health plans and 25% of provider organizations use AI tools in administrative workflows. Payers are ahead, so a provider’s prior auth or appeal may well meet automation on the other side.

The healthcare AI agent use-case table

Use this to scope a pilot. The tier, PHI exposure, and sign-off columns are my judgment.

Coding and patient messages are the easiest rows to scope wrong. Both slide toward auto-send when the approval step is a habit rather than a hard step in the tool layer. The human-in-the-loop design guide shows how to enforce it in code. Prior auth has its own post: AI agents for prior authorization.

What does HIPAA require of a healthcare AI agent?

The current HIPAA Security Rule is the law today, and it covers an agent the moment the agent touches electronic PHI. As HHS summarized in the Security Rule proposed rule, the rule “requires a regulated entity to conduct repeated risk analyses that consider any changes to its environment or operations.” Adding an agent is such a change.

The proposal, published January 6, 2025, would go further. HHS said it would expect “AI software used to create, receive, maintain, or transmit ePHI or that interacts with ePHI, including where ePHI is used to train the AI software” to be listed in a written technology asset inventory. It would also make nearly all safeguards required rather than addressable, and require a business associate to report activation of its contingency plan within 24 hours.

It is still a proposal. Comments closed March 7, 2025, and the latest Unified Agenda entry moved it to Long-Term Actions with final action targeted for July 2027. My recommendation: build the inventory now. The risk analysis duty already applies.

The trouble is how many places PHI lands. An agent writes it into prompts, retrieved documents, tool results, observability traces, evaluation sets, and vector indexes. Every store needs security controls: an access rule, encryption, and a retention period. Any outside vendor that hosts or processes that PHI for you also needs business associate agreement (BAA) coverage for the service you use. Vendor checks are in the HIPAA and SOC 2 checklist, provider retention in zero data retention for LLMs, and logging without copying chart text in the agent audit trail guide.

When does a clinical AI agent become an FDA-regulated device?

FDA’s revised Clinical Decision Support Software guidance, current version dated January 29, 2026, keeps four criteria for software that is not a device. It must not analyze medical images or signals. It must display or analyze medical information. It must support or provide recommendations to a health care professional about prevention, diagnosis, or treatment. And it must let that professional “independently review the basis” for the recommendation.

The 2026 change that matters for agents: when only one recommendation is clinically appropriate and the other criteria are met, FDA intends to exercise enforcement discretion. One example is a single recommended care pathway for chronic low back pain that a clinician reviews and acts on.

Two limits stay hard. Software for “a critical, time-sensitive task or decision” fails criterion 4, because the clinician lacks time to check the basis. FDA ties that reading to automation bias, “the propensity of humans to over-rely on a suggestion from an automated system.” FDA says its device policies still apply to functions intended for patients or caregivers.

Decision rule: assume device territory and get regulatory counsel if the agent reads images or waveforms, sends a recommendation straight to a patient, or supports a time-critical decision. In that territory, FDA’s AI-enabled device software functions guidance describes what a marketing submission should contain. It is still a draft, issued January 2025.

Transparency and nondiscrimination: HTI-1 and Section 1557

Two federal rules reach AI a provider buys or deploys, even when it is not a device.

HTI-1 decision support transparency. The ONC certification criterion at 45 CFR 170.315(b)(11) requires certified EHR developers to expose source attributes (a model card covering intended use, training data, validation, and fairness) and to apply intervention risk management. Both duties cover only predictive interventions the developer supplies as part of its certified Health IT module. A model you or another vendor add on your own is not covered, though the EHR must let you record its attributes. The HTI-5 proposed rule, published December 29, 2025, would remove both, citing “no publicly available evidence” of positive impacts on patient care. As of October 15, 2026, I found no final rule in the Federal Register, so HTI-1 still stands. Put the model card questions in your contract now, because the regulatory floor may go away.

Section 1557 nondiscrimination. Under 45 CFR 92.210, a covered entity has “an ongoing duty to make reasonable efforts to identify” patient care decision support tools that use race, color, national origin, sex, age, or disability as inputs, and must make reasonable efforts to mitigate the discrimination risk of each. An agent that ranks referral urgency using age is in scope. Keep a register with the mitigation for each.

State AI laws: the agent can prepare or approve, but not deny

State law is converging on one pattern. Automation may help prepare or approve. A licensed human owns any denial.

California’s SB 1120 applies to health care service plans and the entities they work through. An AI tool used in utilization review must base its determination on the individual’s clinical history and circumstances, not solely on a group dataset. It must be open to regulator inspection and periodically reviewed, and it “shall not deny, delay, or modify health care services” based on medical necessity.

Washington followed. SB 5395, approved March 23, 2026 and effective June 11, 2026, says “Artificial intelligence shall not be the sole means used to deny, delay, or modify health care services.” Carriers that wrote at least 1% of Washington accident and health premiums must also report, among their prior authorization data, the percentage of total denials aided by AI, first due October 1, 2026. So log which prior auth denials an agent touched, from day one.

CMS’s WISeR model mirrors this in traditional Medicare. It uses AI and machine learning to review selected services in six states from January 1, 2026 through December 31, 2031, and CMS states that “all recommendations for non-payment are determined by appropriately licensed clinicians.”

On the provider side, California’s AB 3030 requires generative AI patient messages about clinical information to carry a disclaimer and instructions for reaching a human, unless a licensed or certified provider read and reviewed the message.

Federal action has not wiped these out. Executive Order 14365, signed December 11, 2025, created a task force whose “sole responsibility shall be to challenge State AI laws” that conflict with its policy. The order does not itself invalidate any state law. Design for the strictest state you serve.

What the evidence says: ambient scribes as the test case

The first randomized trial of ambient scribes, published in NEJM AI on November 26, 2025, assigned 238 outpatient physicians in 14 specialties to Microsoft DAX Copilot, Nabla, or usual care from November 4, 2024 to January 3, 2025. Nabla users saw a 9.5% decrease in time-in-note versus control (95% CI, 1.8% to 17.2% decrease). DAX users showed no significant change. Clinicians reported clinically significant inaccuracies “occasionally.”

Two details matter. DAX was used in 33.5% of 24,696 visits and Nabla in 29.5% of 23,653, so the result came at roughly one-third adoption. And time-in-note is an Epic log metric. The trial’s preprint notes that it does not count editing inside the scribe vendors’ platforms, so the measured savings may be overstated. It is not total documentation labor.

A hypothetical worked example. Say a 20-physician group works 4 clinic days a week, 46 weeks a year: 3,680 physician-days. Assume 90 minutes of measured EHR note time a day. At the 9.5% point estimate, each day cuts 8.55 minutes of EHR note time, about 524 hours a year for the group (26 per physician). At the low end of the interval, 1.8%, it is about 99 hours. At the high end, 17.2%, about 949. With a 60-minute baseline, the point estimate drops to about 350 hours. These are reductions in EHR note time only. Deduct time spent reviewing and editing in the vendor’s platform before you claim net savings, and plan on the low end until your own controlled pilot says otherwise.

The failure mode is signing notes without reading them. Detect it with a sampling audit. To have a 95% chance of catching at least one error when the true error rate is 2%, review 149 random notes against the audio. At 5%, 59 notes are enough. Set your tolerance with the hallucination budget method.

Pre-deployment compliance checklist (pass/fail)

Run this before any agent touches PHI in production. The thresholds are mine, not a regulation’s.

Any failure on checks 2, 5, or 7 blocks go-live. Two or more other failures mean a Tier 1 pilot only, until fixed.

Bottom Line

Start with Tier 1 and Tier 2 work, with real sign-off on everything that leaves the building.

Keep the agent away from any decision that says no to a patient. Every 2026 rule change points that way.

Run the checklist before go-live. No evidence column, no launch.

FAQ

Is the new HIPAA Security Rule final?

No. The proposed rule was published January 6, 2025, and the latest Unified Agenda lists a July 2027 target. If finalized, HIPAA’s general rule allows 180 days after the effective date to comply.

Do California SB 1120 and Washington SB 5395 apply to hospitals and clinics?

They are written for payers: SB 1120 for health care service plans, SB 5395 for carriers and Washington’s public employee plans. SB 1120 also covers tools a plan uses through contracted entities, so a vendor building review tools for a California plan inherits its terms.

When do payer prior authorization APIs go live under CMS-0057-F?

API compliance dates generally begin January 1, 2027, and exact dates vary by payer type, per the CMS fact sheet. Decision timeframes of 72 hours for expedited and seven calendar days for standard requests applied from 2026 for most impacted payers, along with specific denial reasons. Drug prior authorizations are excluded from both.

Does Section 1557 apply to non-AI tools too?

Yes. The definition covers “any automated or non-automated tool” supporting clinical decisions, so paper scoring rules count. The duties applied within 300 days of July 5, 2024, which was May 1, 2025.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect