Insurance organizations are eager to unify their data, and it's difficult to think of a time when cloud-first data tooling was in a better or more competitive state. The industry has learned hard lessons from the “big data” era of technology hype, and making the scramble to AI capability easier is a compelling reason on its own to have unified data.
However, these sorts of projects have a track record of failure – not from lack of vision, drive, or even capital but from execution missteps endemic to insurance and other highly regulated industries.
This doesn’t have to be the case. With the right preparation framework, many pitfalls can be avoided before beginning major data projects. Better still, insurers can follow the advice that their agents and brokers would give customers. They can incur minor continuing costs (“premiums”) to protect against various catastrophes like blown deadlines or cost overruns.
Fragmented and Legacy Systems aren’t going anywhere
Many insurers are rightly proud of their history, but it inevitably bleeds into aging technology stacks. It’s a self-reinforcing loop, as the systems implemented 10, 20, or 30 years ago are relied upon and influence multiple data domains. Internal IT and development team members keep their skillset focused, since the business problems they solve are hard enough, and eventually, job descriptions – including for new hires – require experience in the legacy tech.
But even a longstanding system can’t expand everywhere. Insurance is an industry built on partnerships and connections among independent entities, with dozens of layers and players for an average policy. Similarly, when organizations grow through acquisitions, these software and data systems are typically retained in place, keeping fragmentation alive.
Buying a Data Unification Policy
Before embarking on a specific three-phase approach, there are foundational concerns that must be addressed. These require both concrete costs (time and money) and the ability to make clear decisions about specific topics.
- Compliance and Security: Do we know how to store data from partners and internal systems? Control access to it from the start, and derive value from tokenization and masking?
- Data Governance: Do we have a plan to do more than meet regularly to discuss governance? Are we ready to apply software-based controls to govern our data?
- Data Reliability: Will we observe how our data flows to continuously test its quality? When it’s broken, will we know how and where to fix it?
- AI Readiness: Do we have real use cases we’re targeting that we’re legally allowed to pursue? Do we know how to control costs and explain what AI tooling is doing?
These topics aren’t just technical, but there’s a clear and significant danger in staying away from the weeds. It’s worth coming to clear decisions about tools, architecture, and approaches. Even concerns like DevOps, cost optimization, and monitoring dashboards should be considered to a minimum level of granularity.
The other component of this equation, no less important, is achieving buy-in from business, risk, and finance teams. The former will be contributing significant resources to governance in any unification project worth doing. Knowing how to explain the value here is critical. Having a clear technical plan to show non-technical people will ensure their confidence.
Accepting Fragmentation and a Federated Model
It’s tempting to fantasize about a single monolithic data environment, perfectly clean and united, where truth and definitions are constants for every consumer. At a small enough scale (and low enough loss ratios), this is possible.
For the rest of us, however, the latest advances in cloud data technology offer a federated approach, where claims, billing, brokerage, and other teams keep their fragmented systems relatively in place. Instead of copying data into the monolith, queries with centralized compute but distributed connections are used. Fewer pipelines must be developed and maintained, data duplication is reduced, and each line of business has the ability to join the unified world at its own pace.
This approach only works if you’ve paid the premiums for your data unification policy. Without centralized identity and access management, security and governance become unsustainable. Without a unified governance catalog for federated data sources to feed metadata into, knowing where to get the data for a use case becomes difficult or impossible. Without data reliability, the few pipelines that must be built for each line of business risk failure, and a unified analytics layer will find its value diminished.
The federated approach can, at first glance, seem like a risk: analytic workloads here can be more expensive than those built on top of a monolith. This approach reflects how most insurance environments actually function. However, by mitigating risk upfront with dedicated planning and a pragmatic architecture, data unification moves from fantasy to reality.
