Given its immense and growing capabilities, more and more healthcare organizations are, understandably, looking to implement AI. Adoption is up, with 75% of US health systems using the technology in 2026 – a steep increase from 59% last year. But as AI moves into most corners of American healthcare, the industry must take a step back. AI use for the sake of AI use is not the goal. Rollout of the technology must be targeted, especially as trust in AI is called into question.
This is true across the entire sector, and I've seen it up close across the field I operate in: contested medical review and health insurance disputes. Here, it's vital to insist that AI only be used where it meaningfully addresses a problem and, crucially, that it never makes an important medical decision.
Where AI can help: The No Surprises Act case study
Take the No Surprises Act (NSA) Independent Dispute Resolution (IDR), for example. When the No Surprises Act took effect in 2022, it set off a crescendo of disputes between insurers and providers, which reached 394,140 monthly cases as of July this year. With hundreds or even thousands of pages of documents per case, independent dispute resolution entities (IDREs) have ramped up efforts to handle the sheer volume of evidence. But such enormous numbers remove the adjudication process from the realm of human capacity alone.
That's an area where AI can play an important role, keeping costs and capacity crunches under control. By trawling through documents, pulling out necessary information, and comparing and contrasting evidence from payers and providers, AI can strip away a lot of the administrative burden that takes up so many hours, allowing arbiters and experts to make informed decisions on more cases.
Where AI doesn't belong: Final medical decisions
Equally, though, it also demonstrates a perfect example of where AI simply doesn't belong: it should never make the actual judgment in any insurance dispute. That decision must always remain with a human expert. Just because AI can make the call, and might do it quicker, doesn't mean it should. Taking that step would undermine something even more important: trust.
Rising scrutiny and accountability
Maintaining and building that trust is even more crucial at this moment, because scrutiny of AI's use in IDR is undoubtedly ramping up. In September, the office of New Jersey Rep. Frank Pallone sent letters to six IDREs, asking a series of questions about their operations. One of those questions zoomed in on AI in the IDR process.
This scrutiny is reasonable. These processes carry real implications for insurers, doctors, and hospitals. An IDR decision is a real healthcare decision that determines what insurers pay for procedures, and how much providers receive. In my view, that is enough reason to keep AI out of the direct medical decision-making altogether.
Pallone's letters hint at a vital question many have about all aspects of health AI – one that extends far beyond the healthcare insurance space: is it making decisions about my healthcare without my knowledge?
Growing skepticism across healthcare
Because the skepticism is rising across the board. Federal initiatives such as Medicare's AI-led WISeR program and new patient chatbot initiatives, for example, have come under fire for making key decisions and even prescribing drugs to patients. Irrespective of whether these uses of AI are appropriate, there's no denying that narratives around them are shifting. They are proof that building trust and being targeted with the technology's use at this moment in time is vital.
What patients want: Transparency and control
There is clearly still appetite for AI. Patients themselves are increasingly turning to chatbots for medical advice alongside that of their doctor. What matters is the execution. You can see that in polling by the Pew Research Center, which suggests that an overwhelming 72% of Americans believe it is very or extremely important to know when AI is used in their healthcare – including in administrative decisions, or even note-taking. It's clear that patients see a difference between AI nebulously injected into every aspect of their regular care, and targeted, careful use that adds another component, or extra layer of support, to it.
Building trust through transparency
The answer for all organizations, then – from healthcare providers and healthtech firms to governments – is to make trust a top priority. That means hard-and-fast rules about transparency and accountability. For insurers, it means clearing up what prompts are used to guide generative AI, and, in the context of IDR specifically, where they are relying on human experts to make key judgments about payment offers. In some cases, third-party AI audits might be necessary, but whatever the measures, they must be openly communicated to the public.
Crucially, every organization must be targeted with the technology's use. In every application, the use case needs to be clear from the start. Does AI complete the work better than a doctor, a claims specialist, or an underwriter, or just faster? Speed and volume – AI's strengths – should not necessarily be the priority across all areas of healthcare, especially when they come at the cost of accuracy, reliability, accountability and transparency.
A critical juncture for healthcare AI
This feels like a critical juncture for public trust. Moving forward, we must implement the technology only where it meaningfully solves a problem, and never, ever, to make an important medical decision on its own. Indiscriminate use risks permanently damaging trust and, if we let that happen, we'll lose so many of the meaningful improvements AI has the potential to bring to patients across the world.
