Picture a product feature built almost entirely with AI. A product manager uses an AI assistant to turn a business request into user stories and acceptance criteria. Developers hand those stories to a coding agent. QA generates the test suite from the same stories. DevOps asks AI to update the pipeline that deploys it. Every test passes, every check is green, and the change ships on schedule.
Now suppose the original story misread one rule: A limit meant to apply to each coverage was written as applying to the policy as a whole. Every downstream AI faithfully built, tested, and deployed that misreading. Nothing failed, because every step checked its work against the step before it. The first sign of trouble would be a claim paid against the wrong limit. So where, exactly, was the governance?
In 20 years of building software, I have not seen anything disrupt the development and architecture lifecycle the way AI has. It is changing how we work, faster than most organizations can absorb, and that speed carries real risk. Every team is adopting AI in its own way, with its own patterns, tools, and agents, trying to find what works for them. The natural response is to standardize. But is standardization even the answer?
Why insurance raises the stakes
Insurance has very little tolerance for error. It is one of the most heavily regulated and litigated industries, and the cost of getting things wrong goes well beyond a bug fix: regulatory fines, market conduct actions, policyholder lawsuits, reputational damage, and in serious cases, a threat to the license to operate.
Rating logic, underwriting rules, and policy forms are often required to match what an insurer has filed with state regulators, so a code change can quietly become a compliance gap. Decisions that affect policyholders face growing expectations of fairness and explainability. Carriers also operate under regular audits and regulatory examinations that expect documented, accountable approvals.
Regulators are extending those expectations to AI. The NAIC adopted its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers in December 2023, and more than half of U.S. states have since adopted it. The bulletin expects insurers to maintain a written governance program for AI that makes or supports regulated decisions, and regulators can examine that governance during market conduct reviews. Some states go further: Colo. and N.Y. apply their own frameworks, which add specific requirements such as testing AI systems and external consumer data for unfair discrimination.
The same logic should extend to AI that builds the systems making those decisions. If a model's output needs governance, so does the code that encodes a rating rule or a policy limit.
The autonomy ladder
A simple way to think about AI in software delivery is as a ladder of autonomy:
| Level | Name | What the AI does | Who owns the change |
| 1 | Suggest | Offers completions and snippets | A human writes and owns it |
| 2 | Draft | Generates a full change | A human reviews and merges it |
| 3 | Gated approval | Advances change through automated gates, with humans holding defined approval boundaries | Gate rules plus named human approvers |
| 4 | Autonomous | Plans, changes, and deploys with no human approval boundary | The process itself, unless a human owner is explicitly defined |
The last column is the one that matters. As autonomy rises, it gets harder to name who owns the change.
Teams do not always move up this ladder deliberately. Many are already drifting from Level 1 to Level 2 as coding assistants shift from suggesting lines of code to generating complete changes, tests, and pull requests. Once agents can invoke tools, modify multiple components, or satisfy automated merge gates, the workflow is effectively operating at Level 3, even if the organization still thinks of the technology as a developer assistant.
Autonomy accumulates across the delivery lifecycle. An agent that drafts code, generates tests, responds to review comments, and prepares deployment changes may still appear to be operating under human supervision at each individual step. Taken end to end, however, much of the change has effectively been produced and advanced by the agent. Governance therefore has to consider the accumulated autonomy of the workflow, not just the apparent autonomy of each tool invocation.
High-impact insurance systems, the ones that drive underwriting, pricing, and claims decisions, can go as far as Level 3, gated approval, but no further. AI can do substantial work, but any change with material underwriting, pricing, claims, or regulatory impact should not cross the final approval boundary without human review. Autonomy is only half the picture, though. The same level of AI involvement means something very different for an internal reporting tool than for the system that applies policy limits. Governance has to weigh autonomy against the risk of the change itself.
Five anchor points for governance
Standardizing tools will not keep up, because teams will keep adopting new ones. What carriers can standardize is expectations: where governance anchors the delivery chain, regardless of which tools a team uses. Five anchor points matter most.
1. Anchor intent with accountability. Stories and acceptance criteria are the root of everything downstream, and if the source of truth is wrong, nothing after it can fix it. For high-impact rules such as rating, limits, underwriting, and claims payment, a named person or role with business authority should approve the requirements before any AI builds from them.
2. Keep verification independent of generation. Tests generated from the same AI-written stories as the code cannot catch a misread requirement. The strongest safeguard is a body of end-to-end tests that humans own. QA teams build them from the knowledge of underwriters, actuaries, and other business and technical experts, and from how existing systems actually behave, which is often the most reliable record of the real business rules. They do not share the code's AI-written origins, so they can catch errors that generated tests would pass.
3. Tier approval by system risk. Classify systems first, with rating and claims payment logic at the top and internal tooling at the bottom. Then let allowed autonomy and approval requirements follow the tier. If AI can modify the pipeline, it can weaken the very controls meant to catch problems, so pipeline changes need separate review. In practice, the most effective human gate is not reviewing every generated line of code, which quickly becomes a rubber stamp. Instead, it is requiring architects or subject matter experts to approve the artifacts with the widest blast radius, such as API specifications, contracts between systems, and CI/CD configuration, before anything merges to main or reaches a shared environment.
4. Preserve AI provenance with the change. Provenance means recording the origin of a change. Real work is often mixed: a human writes some parts, an agent writes others, and the human edits the result. Rather than a binary "AI-generated" tag, capture provenance metadata showing which tool or agent was involved and to what degree, through pull request labels, commit trailers such as "AI-Assisted: yes," and tags on stories, tests, and pipeline changes, each linked to the request that started it. This lets teams route AI-involved changes to underwriting, pricing, claims, or regulatory controls for the right level of review, demonstrate that certain tests remain human-owned, and show auditors not just what changed but how. Capture it at the point of creation: reconstructing it later is difficult, incomplete, and often unreliable.
5. Make the chain traceable, at build time and at run time. Any deployed change should trace back through test, code, and story to the person who approved the original intent. The running system must be traceable too: generated code should carry correlation IDs, structured logs, and business events that record which rule or limit was applied to which policy or claim, without capturing sensitive policyholder data. AI tools rarely add these unless required, so they belong in standards and templates that generated code must meet, with pipeline checks that fail when they are missing. When a regulator, auditor, or policyholder asks why a decision was made, the answer should be reconstructable from the record, not from memory.
Where to start
For carriers wondering where to begin, four steps make a practical 90-day start:
- Inventory where AI is already used across product, development, QA, and DevOps.
- Tier systems by risk and set the maximum autonomy allowed for each tier.
- Name accountable business owners for high-impact requirements, and keep human-owned tests for the most critical rules.
- Make provenance metadata, correlation IDs, and decision logging required standards for generated code, enforced in the pipeline.
The speed can come from AI
AI is not going away, and carriers should not be overwhelmed by it. It can dramatically speed up how we build software, but it can also be confidently wrong, and it does not understand our business the way our underwriters, actuaries, and engineers do. That domain knowledge is exactly what governance should anchor to. Teams will keep changing tools, so standardize the anchor points instead. When someone asks why a decision was made, "the AI did it" should never be an acceptable answer. AI can bring the speed. The accountability stays with people.
