The insurance industry has spent the last three years asking one question: where can we apply AI?
It's time to start asking the harder one: where shouldn't we?
We've seen what AI can do in underwriting, claims triage, fraud detection, and customer servicing. The efficiency gains are real. The speed is real. But as we embed AI deeper into consequential decisions — decisions that determine whether a claim gets paid, whether a risk gets written, whether a customer gets flagged — we're walking into a compliance gap that most organizations haven't fully reckoned with yet.
The definition of compliance is changing underneath us.
Traditionally, compliance in insurance was about people and processes following rules. Regulators asked: did you follow the procedure? Was the policy applied correctly? Can you show me the file?
Those questions still exist. But they're being joined by a new set:
- Why did the AI make this decision?
- What data trained it, and was that data appropriate?
- Can the decision be explained to the customer who was denied?
- Can it be audited three years from now when a regulator comes asking?
- When something goes wrong, who is accountable — the model, the vendor, or the carrier?
These aren't hypothetical questions. They're already landing on legal and compliance desks across the industry. And most AI systems aren't built to answer them cleanly.
Here's the tension nobody talks about openly.
The more we automate decisions with AI, the more we need infrastructure that is not AI to govern those decisions.
Compliance requires consistency, predictability, and an evidence trail that holds up under scrutiny. In many cases, the right tool for that job is deterministic — rules engines, governance frameworks, workflow controls, immutable audit logs, human checkpoints at defined thresholds. These aren't legacy artifacts to be replaced. They're the architecture of accountability that AI, by its probabilistic nature, cannot fully provide on its own.
This isn't a limitation to be embarrassed about. It's an architectural reality to be designed around.
The Auditor Cannot Be the Accused.
You cannot use AI to audit AI.
Yet most organizations today are doing exactly that — monitoring AI models with more AI. If the original model has a flaw, the auditing model likely carries the same one. That's not oversight. That's a mirror.
We've already seen AI fabricate citations — confidently, cleanly, and completely wrong. Now imagine that happening in a claims denial or an underwriting decision, where the AI made the call and wrote the audit trail.
Regulators will ask for explainability reports that were produced independently — not one AI explaining another. That documentation needs to be deterministic, traceable, and human-readable. Built without AI. Full stop.
The next wave of RegTech won't be smarter AI. It will be AI governance.
While carriers and insurtechs race to deploy more models, there's an equally significant — and arguably less crowded — opportunity in building the layer that sits above those models. The technology that ensures every AI-driven decision is transparent, explainable, documented, and defensible. The systems that answer the regulator's question before the regulator asks it.
Some of this already exists in nascent form. Model risk management frameworks borrowed from banking are making their way into insurance. Explainability requirements are starting to appear in state regulations around algorithmic underwriting.
The organizations that get ahead of this won't be the ones with the most AI. They'll be the ones who built the governance layer early — and can demonstrate it when it matters.
A word to operators.
If you're running an insurance business and you've deployed AI in any customer-facing or claims-facing capacity, ask your team this week: if a regulator asked us to explain the last 1,000 decisions this model made, could we do it? And could we prove that explanation wasn't itself generated by another AI?
If the answer to either question is uncertain, that's your compliance gap — and it's growing faster than most people realize.
The competitive advantage in this next phase won't come from adding more AI. It will come from making the AI you already have trustworthy enough to defend.
