When AI gets something wrong, who bears the loss?
For the past few years, that question has mainly appeared in discussions of AI governance and legal liability. But as AI moves into core business processes—and as agents begin to execute tasks autonomously—another market is being forced to confront the same question: insurance.
An IT Pro article on AI insurance notes that a new category of coverage is beginning to emerge around AI-related risks. These may include financial losses caused by hallucinated advice, algorithmic discrimination, copyright infringement, models that fail to meet promised performance levels, and even losses caused by autonomous AI agent actions.
This is not yet a mature or standardized insurance product. Coverage is still scattered across standalone policies, endorsements, and exclusions. But the more important development is not simply that insurers are beginning to cover AI. It is that once insurers agree to absorb losses caused by AI, they must answer a harder question:
When an agent can act autonomously, is the insurer underwriting model risk, system risk, or task-governance risk?
What kind of AI deserves to be insured? That question reveals a deeper shift: AI governance is becoming part of the infrastructure of insurability. And it may, in turn, change how companies govern AI. As agents gain greater autonomy, a company’s ability to show that their risks can be understood, constrained, and traced will increasingly shape whether those systems are insurable.
For insurers willing to underwrite AI risk, that challenge is also becoming a market opportunity.
1. From Risk Governance to Insurability
In mature risk markets, insurers first ask whether a risk can be identified, controlled, and estimated. Only then can they decide whether to insure it and at what price.
AI is beginning to enter that process.
Gartner predicted in 2026 that by 2030, property and casualty (P&C) insurers may require robust AI risk controls as a condition for providing affirmative AI liability coverage. Companies with stronger AI governance and risk-management capabilities may not only find it easier to obtain coverage, but may also receive more favorable pricing.
That means companies building AI governance will need to answer more than “How do we manage AI?” They may also need to answer: Can we prove to an insurer that we actually have AI under control?
This is not a paper exercise. An AI policy, a set of ethical principles, or even an AI governance committee is not enough to demonstrate that risk is truly under control. What insurers should care about is whether those governance practices have become operational mechanisms before an incident occurs—and what, in practice, the company has changed because of them.
To assess AI risk, insurers will need to know which models a company uses, which decisions those models influence, and who is responsible for those decisions; whether automated actions are logged; whether high-risk situations trigger human intervention; whether model drift and anomalies are detected; and whether incidents can be reconstructed after something goes wrong.
As one expert quoted by IT Pro put it, governance that exists only on paper, without being translated into operational mechanisms, will not satisfy a prudent underwriter.
AI governance is therefore acquiring a function that has received far less attention: it is becoming part of the infrastructure of AI insurability.
2. What Insurers Underwrite Will Extend Beyond the Model
There is still an unresolved problem. Much of today’s AI risk management remains model-centric. Companies maintain model inventories, test accuracy, monitor bias and model drift, and preserve output records. For generative AI, hallucinations, discriminatory behavior, and incorrect outputs can all create direct financial loss.
But once AI becomes an agent that can execute tasks autonomously, the nature of risk changes. An agent may correctly execute a task that has already become irrelevant. It may continue pursuing the original objective after new information changes the situation. Or every individual action may comply with the rules while the combined result still creates an outcome the company cannot accept.
At that point, model capability alone is no longer enough to assess the risk. The exposure insurers need to understand begins to extend from model risk to task-governance risk.
The underwriting question therefore becomes:
Under what conditions is this AI allowed to do what it does? And if the context changes, will it know when it should stop?
For companies, that raises another question: how do they establish the insurability of an AI agent?
3. Agent Insurability Requires Evidence
Imagine a company that allows an AI agent to process customer refunds autonomously. From a model-governance perspective, we can test whether its answers are correct, whether data is secure, and whether its outputs show bias. But an insurer may also need to know:
How much money can the agent refund? When must it reassess what the customer actually needs? If suspected fraud, a new complaint, or an abnormal transaction appears, is the original task still valid? Who determines whether the agent may continue, pause, or hand the case to a human? And what evidence shows that the task was actually completed as intended?
These questions go beyond conventional model governance. They point to four task-governance mechanisms that can also serve as underwriting evidence:
Task design — defines what the AI is actually authorized to accomplish. For an underwriter, it helps establish where the exposure begins and where it ends.
Permission boundaries — define which data, systems, assets, and actions the AI may access or change, and how far its authority extends. These boundaries directly shape the scale of potential loss.
Understanding loop — determines whether new signals cause the system to reassess the task itself rather than simply continue executing the original objective. This is critical when a change in context should trigger a pause, escalation, or human intervention.
Outcome verification — defines what counts as task completion: merely finishing a process, or producing the business outcome that was actually intended. It also creates the evidence needed for auditability and traceability.
Companies may think of these mechanisms primarily as ways to control agents. From an insurance perspective, however, they serve another purpose: they are evidence that an agent’s risk can be understood, constrained, and traced.
4. Insurance Is Becoming an External Constraint on AI Governance
Internal AI governance has a familiar problem: governance rarely generates revenue directly. Under pressure to move faster, increase efficiency, and deliver business results, it can easily deteriorate into policies, checklists, and compliance documents.
Insurance may change the incentives for companies to govern AI well.
A 2026 AI-insurance study accepted by Harvard Data Science Review points to a fundamental difference between insurers and ordinary AI assurance providers: insurers actually bear the cost of AI-related claims. That gives them a direct economic incentive to demand more rigorous testing, monitoring, and validation—and, through premiums and underwriting conditions, to encourage safer AI governance.
That mechanism is not new. Fire protection, automotive safety, industrial risk controls, and cybersecurity have all been shaped by insurance pricing and underwriting requirements, gradually turning safety practices into standards that markets can verify.
AI may now be entering a similar stage. If a company cannot explain an agent’s task boundaries, operating permissions, intervention mechanisms, outcome records, and post-incident traceability, it may face more than regulatory scrutiny. An insurer may simply conclude:
I cannot price this risk.
Or:
I can insure it—but at a higher premium, with lower limits, and with certain autonomous agent actions excluded.
Once AI governance capability begins to affect whether a company can obtain coverage, the terms of that coverage, and the premium it pays, governance has entered the company’s risk pricing. That may be one of the most important futures for AI insurance.
Conclusion: Are Insurers Ready to Judge AI Insurability?
Discussions of AI liability often end with a familiar principle: no matter how advanced AI becomes, humans remain ultimately responsible.
The principle is sound. But as agents gain greater autonomy, the harder question is no longer whether someone can be found to bear responsibility after the fact. It is whether the company can demonstrate in advance that human responsibility has been translated into operational control mechanisms inside the system.
Who defines the task? Who configures the permissions? What signals require the system to reassess? Under what conditions must it stop or hand control back to a human? And how do we prove that the AI completed the right task—not merely that it executed correctly?
These questions once belonged mainly to AI governance. AI insurance is turning them into underwriting questions.
That could mark an important shift. Once the insurance market begins to price AI governance capability, governance is no longer something a company simply believes it “should do.” It may increasingly determine whether the company is entitled to grant an agent greater autonomy, whether outsiders can trust its AI, and whether the resulting business risk can be transferred.
What ultimately differentiates enterprise AI capability may therefore be more than model strength or the number of tasks an agent can perform. It may be whether a company can prove that, even when AI begins to act autonomously, the resulting risk remains understandable, controllable, and insurable.
Seen from the other side, insurance in the AI era is not only about covering losses caused by AI. It is also about assessing whether a company can turn AI risk into something understandable, priceable, and subject to intervention.
That brings the question back to insurers themselves: as risk moves from models to autonomous agents, do insurers already have the capability to judge whether those agents are insurable? If not, AI insurance will struggle to develop beyond the concept stage.
That capability is not only a prerequisite for AI insurance to become a real market. It may also become one of the insurance industry’s next essential areas of expertise.
