This article is the second of three parts. This first part is here.
The question is not whether intelligent machines can have emotions, but whether machines can be intelligent without emotions – Marvin Minsky
Artificial intelligence, particularly generative and agentic systems, has altered insurance risk not by optimizing processes but by transforming how decisions are formed, executed, and propagated. As AI operates autonomously across tightly coupled workflows, failures that were once local and visible become systemic, invisible, and fast-moving. Errors no longer emerge gradually through human judgment but accumulate unnoticed across the value chain, amplify through feedback loops, and crystallize as financial, regulatory, or reputational risk before intervention is forced.
For insurers, the core question is no longer whether AI can improve underwriting, claims, or service efficiency, but whether the risks introduced by autonomous decision-making, model opacity, concentration, and fragmented governance can be identified and controlled within existing institutional structures. This article focuses on mitigating AI-related risk, outlining the governance, control, and contractual mechanisms required to manage autonomous, probabilistic systems.
Managing the Risks
The same features that make AI powerful also introduce fragility and new risk categories. Speed amplifies error, autonomy removes human checkpoints, and adaptive model behavior can degrade silently until losses accumulate. AI systems, including generative and agentic models, are already embedded in underwriting, pricing, claims, fraud detection, and portfolio management. In these systems, errors can propagate at automated scale, model drift can distort outcomes without warning, and concentration in shared models and platforms can create correlated exposures across firms and markets. Managing these risks requires governance, commercial discipline, and continuing regulatory engagement that differ from traditional technology risk management (see Figure 1).
Figure 1: Managing AI Risks
Governance Imperative
Generative and agentic AI introduce unprecedented risks for insurers, yet governance has not kept pace. Companies are absorbed by rapid technology growth and new capabilities, focusing on building and deploying AI rather than governing it. Effective AI governance requires disciplined algorithm and model management, covering interpretability and auditability across production systems. These standards cannot be sustained without executive ownership.
AI Risk Committee
AI governance at the model and validation level requires a cross-functional AI risk committee with decision-making authority over model deployment, not an advisory body. As AI becomes embedded across the entire insurance value chain, the committee and its subcommittees must work in close coordination. The committee must review models for bias and opacity before deployment, enforce continuous monitoring standards, and retain accountability when automated decisions cause harm.
Establishing AI Leadership
Insurers must formalize AI leadership to make strategic oversight a baseline requirement. The role that meets this need is the chief AI officer, a senior executive with authority to align AI strategy with business goals and enforce consistent governance. An AI center of excellence under this role centralizes expertise, aligns AI initiatives, and enforces accountability to reduce risk and meet regulatory obligations.
Vendor Diversification
Vendor diversification is critical to managing generative and agentic AI risk. Insurers must assess AI providers technically, not treat procurement as commercial exercise, and avoid over-concentration on a single vendor. In periods of geopolitical stress, providers may face state or regulatory action that restricts access to platforms or services, disrupting the insurance operations.
Organizational AI Literacy
AI literacy does not require technical depth across the organization. It requires underwriters, claims managers, and executives to spot failure signals, question automated decisions, and override them when needed. This human judgment underpins all technical controls. Governance fails when people do not understand model limits or risks, or when outputs are treated as authoritative and human review becomes procedural.
Controlling the Machine
As AI systems shift from decision support to autonomous action, control becomes an engineering and governance problem rather than a procedural one. Decisions are executed continuously at speeds beyond human review, collapsing the gap between judgment and consequence. Risk no longer stems from discrete failures but from interaction and scale. Managing such systems requires authority over machine behavior in production, the ability to recognize unsafe autonomy, and early intervention before errors become embedded or irreversible.
Human-in-the-Loop
Human‑in‑the‑loop override is an operational requirement inherent to automated and agentic AI systems, arising from their technical limits rather than preference. Automated and agentic systems act based on their training data, objectives, and delegated authority, and when real‑world conditions fall outside those bounds, decision quality degrades, often without warning. The ability for a qualified human expert to review and override an agent's decisions at defined control points, supported by an organizational culture that encourages such intervention, remains the primary safeguard against silent systemic error, particularly when those errors propagate quickly and are hardest to detect.
Adversarial Red Teaming
Insurers should adopt adversarial red teaming as formal control for AI risk. An internal team independent of model development and deployment should probe production AI systems to identify how they fail rather than confirm that they operate as designed. In practice this includes testing whether claims models can be misled by fabricated evidence, whether pricing models can be manipulated through synthetic applicants, and whether automated damage assessment systems can be induced to produce incorrect outcomes. Testing that is limited to known attack patterns is regression testing rather than red teaming. Effective red teaming focuses on discovering previously unknown failure modes, which are the failures most likely to surface in real‑world operations.
Independent Model Validation
Insurers should treat independent model validation as a core control that complements regulatory audit and counters the natural incentives of model owners to focus on intended performance rather than failure. This requires validation teams with genuine independence from the business units they review, supported by out‑of‑sample testing, adversarial stress testing, and reporting lines insulated from the business units whose models are being validated. The cost of adequate model validation is a fraction of the expected loss exposure from a single ungoverned model failure, whether through pricing error or discriminatory outcomes.
Data Lineage Tracking and Drift Management
Knowing precisely where training data came from, how it was processed, and what rights are attached to it is essential to address IP liability, regulatory exposure, and bias risk at the same time. Without this, the insurer will be building intelligence models on a foundation whose integrity cannot be verified. Insurers need robust, data‑driven schedules for retraining and updating AI systems, with automated triggers that pause models when performance falls below a defined threshold to address drift directly.
Model Output Guardrails
Model output guardrails define the boundaries within which an AI system is allowed to operate. They serve as a standing control that limits the impact of vulnerabilities identified through red teaming. In insurance deployments, guardrails should operate at the content, decision and output levels. This includes constraining pricing outputs to actuarially defensible ranges, enforcing mandatory human review for high‑impact or regulator‑sensitive decisions, and restricting generated customer communications to legally verified positions.
Kill Switch
The kill switch is the operational mechanism by which a deployed AI system can be immediately suspended, constrained, or rolled back when its behavior is identified as harmful, anomalous, or outside the parameters permitted by its governance framework. This is not a conceptual safeguard, but a technical mechanism embedded in production workflows. A functional kill switch includes automated triggers that act when predefined thresholds are breached, such as surges in adverse underwriting decisions, pricing outputs that fall outside actuarially defensible limits, or claims error rates that exceed tolerance levels. Governance requires kill switches that operate at machine speed and authority structures that allow rapid intervention that is proportional to potential risk.
Risks and Coverage
Insurers occupy dual positions as both providers of coverage for AI-related risks and users of AI-enabled technologies within their own operations. AI-related risk and liability implications extend beyond insurers' internal operations to the external threats across the risks they underwrite. AI introduces loss dynamics that differ materially from traditional drivers, and agentic systems require a reassessment of how coverage is defined, how fault is attributed, and how liability is distributed. The nearest relatable risk category to this is cyber risk, not because the mechanisms are identical, but because both introduce systemic, non-linear losses that propagate across insured ecosystems. This exposure requires tight alignment between underwriting intent, policy wording, and operational risk appetite, to ensure that the liabilities insurers accept can be priced and controlled within the boundaries implied by the coverage they offer.
Rethinking Policy Wording
Traditional insurance policy contracts were not designed for a risk environment in which losses arise from the use of autonomous systems by customers across their operations, products, and services. Updating policy wording therefore requires more than incremental change. Insurers must reassess how coverage is defined, how causation and responsibility are attributed when autonomous systems fail, and which categories of AI-driven risk they are prepared to underwrite. AI-specific inclusions and exclusions must be drafted deliberately and precisely, rather than adapted from existing cyber or professional indemnity language. The legal and financial distinction between AI system failure and conventional technical or operational error must be explicit, as ambiguity will inevitably lead to coverage disputes.
AI-Specific Exclusions and Sub-Limits
Traditional underwriting approaches must be extended to address AI-specific risk vectors, particularly those introduced by generative and agentic systems that remain difficult to price reliably. Insurers should use targeted exclusions and sub limits to bound exposure to failure modes such as losses arising from hallucinated or fabricated outputs and cascading agent behavior that propagates errors across systems. Sub limits that cap cumulative exposure from a single automated decision thread or agent-driven process are a prudent portfolio control, especially in early deployments where behavior remains unstable. Coverage should also be restricted where generative outputs are used without mandated human review, where models are retrained or prompted outside approved controls, or where autonomous systems adapt objectives not disclosed at the beginning.
AI‑Native Risk and Coverage Constructs
Insurance coverage for AI must evolve because risk triggers shift from discrete, event‑based, human‑initiated failure to continuous, autonomous behavior operating at scale. Traditional coverage lines such as cyber liability, professional liability, directors' and officers' liability, product liability, intellectual property, employment practices, and regulatory liability were built around predictable failure modes, identifiable human acts or omissions, and linear causation. These assumptions break down with respect to agentic systems that learn, interact, and act autonomously in production. Recalibration therefore requires changing how these existing covers attach and respond. They must be re‑anchored to observable system behavior rather than point failures and structured to distinguish between system failure and intentional interruption of automation. At the same time, the shift in risk source necessitates entirely new coverage constructs to address exposures traditional frameworks were never designed to absorb, such as model failure protection, autonomous decision indemnity, AI‑enabled fraud coverage, AI supply chain liability, AI‑triggered business interruption, and catastrophic AI accumulation risk.
The Customer Obligation
As decision making authority shifts from people to machines, the insurer's obligation to the customer changes in character, and meaningful recourse. Trust in an AI-driven insurance model depends on whether customers can know when automation is at work and are able to challenge outcomes through processes that are independent, effective, and humane.
Explainability and Transparency Disclosures
The obligation of the insurer to explain pricing and underwriting decisions in terms that a customer can understand is both a regulatory requirement and a minimum standard of fairness. Transparency disclosures about when and how AI is used, including key data points that affect pricing, and what recourse is available when something goes wrong, are the foundation of informed consent in an AI-driven market.
Addressing Protection Gaps
AI increasingly enables granular risk segmentation and personalized pricing, leading to a decomposition of the insurance risk pool. Consequently, lower-risk individuals benefit from reduced premiums, while higher-risk individuals are priced closer to full actuarial cost, which may even make insurance unaffordable to some. The resulting protection gap disproportionately affects low-income households and small businesses, which often carry higher structural risk and lack the resources to absorb higher premiums or meet AI-related risk requirements. To address this, regulators and policymakers may need to define limits on permissible risk granularity and pricing personalization to preserve insurance's social function and economic resilience. As the coverage of last resort to absorb catastrophic AI failures, public-private risk pools, including government-backed schemes, may be necessary for AI-related systemic events. Such mechanisms function as the reinsurance equivalent for systemic AI exposure and complement regulatory constraints on pricing and underwriting practices.
Customer Appeal Mechanisms
A customer appeal mechanism is the operational and legal pathway through which an individual, who was adversely affected by an automated insurance decision, can challenge that decision and obtain a review that is substantively independent of the system that produced it. The effectiveness of the mechanism depends on design rather than mere availability. An effective functional appeal mechanism requires that the customer is informed, in clear terms, of the factors considered in making the decision. Upon appeal, the earlier decision must be reviewed by a qualified human with authority to change the outcome. For insurers, appeal mechanisms are the procedural complement to explainability obligations.
Build for Scrutiny
The regulatory environment for AI in insurance has moved beyond principles and consultation. Regulators across many jurisdictions have published governance and audit requirements for AI systems in use, especially in underwriting or claims. Regulators advise insurers to adopt a risk-proportionate supervisory approach, calibrated to whether a system is customer-facing and to the scale of deployment.
Internal Regulatory Readiness
Internal regulatory readiness requires creating an inventory of all AI systems in production. These should be audited for bias, including sensitivity analysis, error rates, and plain language explainability. It also requires audit trails that can withstand regulatory examination. Regulatory requirements now specify action thresholds at which a model must be remediated or disabled, supported by full validation documentation available for internal and third-party audits.
Immutable Logs
An immutable log is a record of AI system activity that, once written, remains fixed and tamper-proof, preserving a complete and authoritative account of the actions and decisions it records. Immutable logs capture model inputs, decision outputs, human overrides, and timestamps in a sequenced and verifiable form. In insurance context, it is a record of how AI systems influence underwriting, pricing, claims handling, and customer decisions. The requirement for immutability arises when an automated decision is later challenged by a policyholder, a regulator, or a court. In those situations, insurers must demonstrate exactly what data was used, which model version was active, what decision was produced, and whether any human intervention occurred, based on contemporaneous records rather than post hoc explanations.
Proactive Regulatory Engagement
Proactive regulatory engagement goes beyond risk management and requires deliberate action. For insurers, this means engaging regulators early on specific AI use cases, sharing evidence on model performance, limitations, and failure modes. It also involves testing proposed governance approaches against production systems rather than principles. Effective engagement includes supervisory discussion on model validation, monitoring, and override processes, as well as transparent disclosure of where automation is used in customer-facing decisions. Insurers that engage in this way are better positioned to influence standards toward technically workable requirements and to demonstrate credible oversight grounded in operational reality.
Governing AI in Production
AI adoption in insurance has moved beyond experimentation, but governance maturity is yet to keep pace. While AI is deployed across the insurance value chain, controls do not operate effectively because they are not adequately resourced, and AI systems that are not governed at the model level remain an unmanaged risk. Responsible AI deployment requires governing the model, not just monitoring outcomes. This includes accountable leadership and intervention capabilities that allow automated decisions to be challenged or withdrawn. It also requires legal structures, including coverage design and policy wording that align with autonomous decision-making. Customer protection depends on ensuring that speed and scale do not outrun obligations to explain decisions, price fairly, and provide effective appeal. As AI deployment is already underway, the issue is more about whether governance is sufficient to avoid regulatory intervention, litigation, and loss of trust.
