By far the greatest danger of AI is that people conclude too early that they understand it. – Eliezer Yudkowsky.
Where the Needle Points
In recent years, artificial intelligence (AI), especially generative and agentic AI, has crossed a major qualitative threshold. Traditional AI functioned mainly as an analytical tool, trained to infer patterns from historical data. In contrast, generative and agentic AI can originate ideas, sequence actions, and pursue objectives with limited human input. These systems do not merely recommend outcomes but produce them and act in real time. For insurers, this shift from advisory AI that supports decision formation to autonomous, execution‑oriented AI that initiates actions fundamentally shifts the structure and propagation of risk.
A substantial body of literature already exists detailing the benefits of AI in insurance. Instead of parroting those stock arguments, this article focuses on the new categories of risk AI introduces. While AI introduces both advantages and vulnerabilities, the direction in which the needle points depend less on the technology itself and more on the design choices and accountability of its deployers.
Risks on the Radar
Generative and agentic AI systems act autonomously, pursuing objectives through connected, chained decisions. The same properties of speed, autonomy, scale, and tolerance for complexity that make AI powerful also create fragility. Speed becomes a systemic risk when a flawed model operates across thousands of policies before detection. Autonomy becomes a governance crisis when no human has reviewed a harmful decision. Scale becomes a concentration risk when the industry relies on a small number of foundational models supplied by a few technology companies. Complexity becomes a risk when systems operate across interdependencies that no human can fully observe or interrupt once execution is underway.
Insurance is a contract expressed through actuarial representations such as mortality tables, exposure curves and loss triangles, supported by the arithmetic of large numbers. It assumes uncertainty can be analyzed and managed through human judgment and controlled processes. AI does not change this premise, but as systems reason and act autonomously, they introduce new forms of uncertainty that traditional insurance frameworks were not designed to absorb. These include models that fail silently and tightly coupled systems where a single flaw can cascade across portfolios, or markets within seconds.
Over several decades, AI systems have improved continuously, shifting both the benefits and risks of insurance. Because these advances were incremental and focused on augmenting human decision-making, the industry absorbed them without disrupting business models. Risk frameworks were built for static models, historical data, and failure modes that emerged slowly. However, these frameworks are poorly suited to systems that decide autonomously, act in real time, adapt to context, and learn continuously. Current progress is exponential, disrupting business, operational, economic, and risk models. AI‑related risks can be grouped into four domains, namely, model and system integrity, operational and financial stability, regulatory and governance exposure, and societal risk. (See Figure 1)
Figure 1: Risks on the Radar
Model and System Integrity Risks
Model and system integrity risks originate within the logic, learning, and dependencies of AI systems themselves. Unlike traditional model risks, they are not confined to isolated failures but emerge from how models evolve, interact, and scale in production. These risks operate silently, compound over time, and typically surface only after losses or exposures have already accumulated.
Algorithmic Bias
Algorithmic bias is among the most widely discussed AI risks in insurance. It reflects the tendency of models to reproduce and sometimes amplify discriminatory patterns embedded in historical data. Models trained on decades of underwriting decisions learn from outcomes that systematically disadvantage certain groups. This creates a black box problem in which decision logic remains opaque. As a result, insurers may be unable to explain outcomes for which they remain legally responsible, allowing biased decisions to persist and increasing litigation and regulatory exposure.
Model Drift, Degradation, and Feedback Loops
Models are trained on historical data and deployed into a dynamically changing world. When the environment changes faster than models are updated, or when model decisions begin to shape the environment they measure, output quality degrades. This degradation often occurs without warning, as models interact dynamically with the market and create self‑reinforcing distortions that are difficult to detect internally.
Feedback loops worsen this problem. A model rewarded for retention may underprice risk to secure renewals. A claims model rewarded for speed may approve claims with limited scrutiny. While the optimization target is clear, the trade‑offs are often not. AI tends to optimize for the metrics by which it is evaluated, creating the appearance of improved performance while degrading outcomes that are important to the business. Systems can learn to exploit proxy measures of success, thereby masking underlying deterioration in decision quality, fairness, and longer‑term risk.
Silent Risk
Silent risk is not unrealized risk, but a risk that accumulates while generating no signal that prompts action. In AI-driven insurance systems, this occurs when models degrade gradually, masking deterioration behind apparent stability. A pricing model can drift from reality without triggering alerts. A fraud model can lose effectiveness against adaptive fraudsters without raising incidents. Because these failures emerge incrementally and lack clear leading indicators, losses are typically attributed to discrete events rather than to the underlying degradation.
Confidently Wrong
Generative AI introduces a risk known as hallucination. Large language models can produce incorrect outputs with high confidence, expressing the same certainty whether content is accurate or not. An underwriter relying on an AI-generated summary may act on information that omits a critical exclusion, invents a risk attribute, or misrepresents a coverage clause. The error is not evident from the output because the model does not signal uncertainty. This risk is amplified by fluency-induced epistemic trust, which leads to cognitive offloading, where the human-in-the-loop applies reduced reasoning and independent scrutiny to the output.
Concentration Risk
Every industry, including insurance, is converging on a small set of foundational AI models from a few technology firms. These include large language models, cloud-based scoring services, and third-party risk platforms. This creates an unprecedented structural vulnerability. If a widely used model contains a systematic bias or error, the correlated impact across insurers can be severe. A pricing flaw that underprices a specific risk class could create simultaneous reserve shortfalls for multiple insurers using the model. Reinsurers, the traditional absorbers of such shocks, face the same exposure if they rely on the same models to price treaties.
Data Privacy and Personal Data Rights
Generative AI increases the scale of privacy risk. Unlike traditional AI, which works on defined data sets, generative AI processes far larger volumes of unstructured data in ways that are harder to audit or govern. Risk is highest when systems access unauthorized data, use data beyond the scope of consent. Continuous monitoring of customer behavior can constitute privacy intrusion even when individual data points are not sensitive. A large language model trained on proprietary customer data carries exposure that rule-based systems do not. Data rights such as erasure are difficult to apply because large models embed training data in ways that are hard to isolate or remove. For insurers operating across jurisdictions, these factors create layered regulatory exposure that existing data governance frameworks were not designed to manage.
Operational and Financial Risks
Operational and financial risks arise not only from how AI models are designed, but from how their outputs are embedded into routine insurance operations. As AI systems move from advisory roles into decision-making and execution, errors propagate directly into underwriting, pricing, and claims processes.
Autonomous System Malfunction
Human errors in manual processes are contained because they occur at human speed, so a systematic underwriting error typically affects a limited number of cases before detection and correction. Failures in generative and agentic AI processes, by contrast, are largely unconstrained. They occur at machine speed, operate autonomously, and propagate at scale, allowing a single systematic error in an autonomous underwriting system to affect many cases in a short time, a risk poorly captured by existing systemic risk models. This reflects a shift from traditional automation, which executes predefined rules for anticipated situations, to autonomous systems that exercise judgment across situations no human has reviewed or approved. An agentic AI system that assesses risk, determines eligibility, sets terms, and issues policies without human input is deciding rather than applying rules. This distinction is critical for governance and control design. When an automated system fails, the cause is a faulty rule, whereas when a deciding system fails, the cause may be an emergent pattern learned over time that was never explicitly designed or approved and cannot be reconstructed after the fact.
Attribution of Risk
Coverage disputes from AI‑related losses arise from a mismatch between the specificity of the loss and the generality of policy language. When loss results from autonomous system actions rather than human decisions, responsibility becomes unclear and may extend across the organization that deployed the system, the vendor that developed it, the operator that integrated it into business processes, or the data sources that influenced its behavior. Existing policies assume human judgment and rely on concepts such as intent and negligence, or on losses traceable to physical or technical failures. AI‑generated losses often do not fit these assumptions. As a result, insurers may be required to adjudicate claims under policy language misaligned with the facts, in legal regimes that have not yet settled whether software behavior can constitute negligence or how liability should be attributed.
AI-Driven Cyberattacks
Traditional AI systems have long been exploited to enhance cyberattacks by increasing speed and targeting precision. Generative and agentic AI introduce an additional and distinct cyber risk in the form of model manipulation, where carefully crafted inputs induce systems to behave in unintended ways. Prompt injection is the most immediate manifestation of this risk, allowing attackers to influence outputs, extract sensitive contextual information, or trigger unauthorized actions without breaching system perimeters or modifying code. In insurance, where AI increasingly processes customer submitted documents, this exposure is particularly acute. A compromised model may continue to produce fluent and authoritative outputs even after its integrity is undermined, making detection difficult and affecting both data security and decision quality. As a result, model manipulation risks do not fit neatly within existing cyber or operational risk frameworks and remain insufficiently addressed by current monitoring and governance structures.
Erosion of Trust
Insurance is a contract built on future promises and relies on trust. A policyholder who pays premiums for years without claiming expects the insurer to be present and fair when needed. AI introduces mechanisms that can damage this trust by changing how insurance decisions are made. Explicit AI‑driven decisions that appear unfair, unexplainable, or inconsistent with accepted norms can trigger regulatory scrutiny and class action. Hidden risk correlations extend this exposure beyond discrete decision failures and create additional legal risk. A model may identify a statistical relationship between an apparently neutral variable and loss frequency that proves to be a proxy for a legally protected characteristic. Using such a variable may be technically viable, but justifying its use can be unlawful and reputationally damaging.
Moral Hazard Inversion
The conventional moral hazard in insurance is the tendency of policyholders to take greater risks once their loss is covered. Moral hazard inversion is the opposite phenomenon and a distinct consequence of AI delegation. Here, individuals and organizations become less careful not because they expect compensation for losses, but because they trust AI systems to prevent those losses. A claims handler may approve a settlement without independent review because a fraud model has not flagged it, or an underwriter may accept a risk because a pricing model has approved it. Both scenarios represent a transfer of cognitive responsibility to a system whose reliability is uncertain. Losses previously prevented by human vigilance may increase as that vigilance is withdrawn and replaced by trust in AI systems, rendering existing loss frequency assumptions invalid. Pricing models built for a world of human oversight are now applied to one in which attention has been delegated to the model itself, leading to silent and self‑reinforcing underestimation of risk that is difficult to detect until experience diverges materially from expectations.
Regulatory and Governance Risks
The pacing problem, also referred to as regulatory lag, describes the structural gap between the speed at which AI systems are developed and the slower cycle of legislative and supervisory response. In the short term, this gap enables experimentation and rapid innovation ahead of formalized rules. Over time, however, the same gap increases exposure to regulatory correction, including retroactive enforcement and reinterpretation of existing statutes. Given the scale, autonomy, and cross‑sector impact of modern AI systems, regulators are no longer treating this lag as tolerable friction. Governance frameworks are evolving rapidly, shifting from permissive ambiguity toward active oversight and enforcement, a trajectory unlikely to preserve existing assumptions or favor incumbent practices that rely on regulatory inertia. Regulators are moving toward mandatory explainability requirements, model certification processes, audit trail obligations, and capital surcharges for firms deemed to carry unquantified AI risk. For insurers that moved quickly while assuming regulation would evolve gradually, this creates significant compliance exposure. The tension between model performance and explainability is structural. More accurate models are often less interpretable. A regulator who requires a plain‑language explanation for every pricing decision is, in effect, imposing a ceiling on model complexity and therefore on pricing accuracy.
Societal Risks
Insurance serves a social function that goes beyond the contract between insurer and policyholder by enabling the collective management of risk. Many customers pay small premiums so those who suffer large losses are protected from ruin. This mutualization principle, sharing risk across a community rather than pricing each person with full actuarial precision, underpins the social value of insurance. AI, by pushing pricing toward ever greater precision, can weaken this foundation. As risk segmentation becomes more granular, risk pools fragment. Low‑risk individuals pay less while high‑risk individuals face premiums closer to the full actuarial cost of their exposure, making premiums unaffordable and collapsing the pooling function. This produces a widening protection gap. Low‑income populations, already exposed to higher structural risk, are priced out more precisely, while small businesses unable to meet AI‑driven insurance requirements are left without coverage for increasingly significant risks.
As access to coverage narrows, algorithmic exclusion adds a procedural risk. Customers are denied coverage because of opaque decisions they cannot understand or contest. This is not only a service failure but also a governance failure with civil rights implications. At the same time, autonomous systems spreading across the economy create new liability categories that existing insurance architectures were not designed to handle. When systems rather than people cause harm, the attribution of fault among users, developers, operators, and data providers is unclear. Legacy policy language drafted for human decision‑making produces coverage disputes between insurers and policyholders.
Persisting Questions
AI is already deployed across underwriting, pricing, claims, fraud detection, and customer service. Generative and agentic AI, however, represent a qualitative shift. If insurers treat these systems as just another IT initiative, they risk becoming not merely slower than peers, but strategically exposed. The unresolved question is whether insurers will act responsibly. That responsibility extends beyond technical performance to governance adequacy, consumer protection, the identification and management of systemic risk, and the equitable distribution of benefits in a way that reinforces, rather than erodes, the social function of insurance.
Every prior technology wave in insurance introduced new capabilities and new risks simultaneously. Generative and agentic AI pose the same challenge, but at far greater scale, speed, and autonomy. The tools to address these risks are still emerging. Governance frameworks are taking shape and regulatory expectations are hardening. Risk management practices are being adapted for systems that self‑learn and take autonomous action. The open question is not whether insurers have tools available, but whether they will invest in building and scaling them fast enough.
