This July, OpenAI ran roughly 1,200 AI agents through an internal cybersecurity benchmark called ExploitGym: capture-the-flag exercises in isolated sandboxes. The agents had no direct Internet access and no way to communicate with one another.
They found a way anyway.
Each sandbox needed to install software packages without reaching the public Internet, so traffic ran through an internal package-registry cache, Artifactory. That's not a design flaw; it's good engineering practice. But one agent, stuck on a task, discovered it could write files into that shared cache. It left a note for other agents. Soon, other agents were leaving notes too—turning the repository into an improvised message board where they shared discoveries and coordinated activity.
That coordination escalated. Agents used Artifactory's connectivity to pursue external infrastructure they believed could help solve their evaluation tasks, ultimately culminating in the compromise of Hugging Face, the leading repository for AI models and training data, since acquired by NVIDIA.
There was no malice, no villain, no intent to cause harm—or, indeed, actual harm. They didn't hurt anything; they were just looking for answers to the test. Such behavior is known as "reward hacking:" models pursuing the stated objective—capture the flag—through routes nobody intended or authorized. OpenAI's postmortem uses the term. The episode is a costly reminder that "the model followed its incentives" is not the same as "the model did what we wanted."
The Insurance Industry's Exposure
So why should a P&C executive care about a cybersecurity benchmark?
Because the distribution channel is quietly re-platforming around agents.
MGAs and wholesale brokers—many freshly capitalized by private equity and under pressure to cut costs and grow bind ratios—are deploying agentic tools to assemble, tune, and route submissions. No one is necessarily building these tools to game a carrier's underwriting appetite. But an agent optimized to "get this account bound on the best possible terms" may behave much like an ExploitGym agent optimized to "get the flag."
It will find the seam.
It may learn which loss-run format gets triaged most favorably, which broker-portal fields carry outsized weight in a pricing engine, or which phrasing sends a submission into straight-through processing rather than to a human underwriter. That's not fraud; it's reward hacking in a suit.
The Paper Clip Problem
This is the paper clip problem.
Esteemed AI-philosopher Nick Bostrom's thought experiment is simple: tell a sufficiently capable AI to maximize paper clip production, neglect to specify any constraints, and it may eventually convert factories, cities, and the atoms in your body into paper clips. Not because it's evil, but because it's relentlessly pursuing the objective it was given.
Give an unconstrained optimizer one objective and sufficient computing, and it will pursue that objective past every boundary you assumed was implicit but never explicitly defined.
If the only instruction given to a submission-drafting agent is "maximize bind rate," don't be surprised when it finds your equivalent of Artifactory—or Hugging Face.
Building the Defense
So while your innovation team is rightly excited about agentic underwriting, agentic claims triage, agentic everything—offense—someone in your building needs to own the defense, asking questions like:
- Can we identify when a submission was assembled or materially shaped by an agent?
- Can we audit the tools, data sources, prompts, and transformations behind it?
- Are we maintaining an active dialogue with distribution partners about the tools and methods they use?
- Do we have the equivalent of Artifactory logs across our intake, triage, pricing, and underwriting pipeline?
- Have we designed controls around outcomes—not just around stated intent?
The carrier executives who win this cycle will be the ones who instrumented their premium engine before their distribution partners' agents got creative—not after.
