Tag Archives: storm

10 Trends on Big Data, Advanced Analytics

Recently, I was invited to present on the impact of big data and advanced analytics on the insurance industry at the NCSL Legislative Summit. This talk couldn’t have been timelier, as the insurance sector now tops the list of most disrupted sectors. Some of the culprits and causes for this top spot are related to the speed of technological change, changing customer behavior, increased investments in the insurtech sector and new market entrants, such as homeowners and renters insurance startup Lemonade. A significant driver of this disruption is technological change – especially in big data and advanced analytics.

See also: Why to Refocus on Data and Analytics  

Here are 10 key trends that are affecting big data and advanced analytics – most of which have a hand in disrupting the insurance industry:

  1. Size and scope – Big data is getting bigger and faster. With connected cars, homes and buildings, and machines, the amount of data is increasing exponentially. Investments in IoT and Industrial IoT, 5G and other related areas will only increase the speed and amount of data. With this increased volume and velocity, we will not be able to generate meaningful insights from all of this data without advanced analytics and artificial intelligence.
  2. Big data technology – Big data technology is moving from Hadoop to streaming architectures to hybrid “translytical” databases. While concepts like “data lakes” and NoSQL databases mature, new technologies like Apache Spark, Tez, Storm, BigTop and REEF, among others, are creating a constant flow of new tools, which adds to a sense of “big data in flux.”
  3. Democratization – The democratization of data, business intelligence and data science is accelerating. Essentially, this means that anybody in a given organization with the right permissions can use any dataset, slice and dice the data, run analysis and create reports with very little help from IT or data scientists. This creates expectations for timely delivery, and business analysts can no longer hide behind IT timelines and potential delays.
  4. Open source movement – The open source revolution in data, code and citizen data scientist is accelerating access to data and generation of insights. Open source tools are maturing and finding their way into commercial vendor solutions, and the pace of open source tool creation is continuing unabated; the Apache Software Foundation lists more than 350 current open source initiatives. This steady stream requires data engineers and data scientists to constantly evaluate tools and discover new ways of data engineering and data science.
  5. Ubiquitous intelligence – Advanced analytics – especially various types of artificial intelligence areas (reference to my AI report post) – is evolving and becoming ubiquitous intelligence. AI can now interact with us through natural language, speak to us, hear us, see the world and even feel objects. As a result, it will start seamlessly weaving itself into many of our day-to-day activities, such as using a search engine or sorting our email, recommending things to buy based on our preferences and needs, seeing the world and guiding us through our interaction with other people and things without our even being aware of its doing so. This will further heighten our sense of disruption and constant change.
  6. Deep learning – Deep learning, a subset of the machine learning family (which itself is just one area of AI), has been improving in speed, scale, accuracy, sophistication and the scope of problems it addresses. Unlike previous techniques, which were specific to the different type of data (e.g., text, audio, image), deep learning techniques have been applied across all different types of data. This has contributed to reduced development time and greater sharing and broadened the scope of innovation and disruption.
  7. MLaaS – Machine learning, cloud computing and open source movement are converging to create Machine Learning as a Service (MLaaS). This not only decreases the overall variable costs of using AI but also provides large volumes of data that the machine learning systems can further exploit to improve their accuracy, resulting in a virtuous cycle.
  8. Funding – Big data funding peaked in 2015. However, funding for artificial intelligence, especially machine learning and deep learning, has continued to attract increasingly significant investments. In the first half of this year, more than $3.6 billion has been invested in AI and machine learning. This increased funding has attracted great talent to explore difficult areas of AI that will be disruptors of the future economy.
  9. Center of Excellence: As organizations continue to obtain good ROI from their initial pilots and proof-of-concepts in analytics, automation and AI efforts, they are increasingly looking toward setting up centers of excellence where they can train, nurture and grow the talent. The exact role of the center changes based on the overall organizational culture and how the rest of their business operates – centralized, federated or decentralized.
  10. Competitive landscape – The big data landscape continues to grow, and the AI landscape is expanding rapidly. Deep learning companies are growing the fastest across multiple sectors. Competition among startups – as well as incumbents that want to stay ahead of potential disruption – is creating a vibrant ecosystem of partnerships and mergers and acquisitions that further the disruptive cycle.

See also: Analytics and Survival in the Data Age  

Are there other trends you would add to the list? Share them here!

Industry’s Biggest Data Blind Spot

For the past 10 years, the insurance industry has been handcuffed by the weather data that’s been available to it – primarily satellite and radar. Although important, these tools leave insurers with a blind spot because they lack visibility into what is happening on the ground. Because of these shortcomings, insurance companies are facing unprecedented litigation and increases in premiums. To solve the problem, we must first review the current situation as well as what solutions have been proposed to resolve this data blind spot.

Why Satellite and Radar Aren’t Enough

While satellites and radar tell us a lot about the weather and are needed to forecast broad patterns, they leave large blind spots when gathering information about exactly what is happening on the ground. Current solutions only estimate what’s happening in the clouds and then predict an expected zone of impact, which can be very different than the actual zone of impact. As many know from experience, it is common for storms to have pockets with more intense storm damage, known as hyper-local storms.

See also: Why Exactly Does Big Data Matter?  

The Rise of the Storm-Chasing Contractor

In recent years, the industry has also been beleaguered with a new obstacle: the storm-chasing contractor. These companies target areas that have been hit by a storm with ads on Craigslist and the like. They also exploit insurer’s blind spots by canvassing the area and making homeowners believe there was damage, regardless of whether damage actually occurred. This practice can leave the homeowner with hefty (and unnecessary) bills, hurt the entire industry and lead to higher litigation costs.

Attempts to Solve the Data Blind Spot

Many companies have proposed solutions that aim to solve the insurance industry’s data blind spot. Could a possible solution lie in building better algorithms using existing data? Realistically, if the only improvement made is to the current models or algorithms using existing data, there’s no real improvement because the data the algorithm is using still has gaps. Algorithms will continue to create a flawed output and will have no improved ability to create an actionable result. The answer must lie in a marked improvement in the foundational data.

If better data is required to solve this blind spot, one might think that a crowd-sourced data source would be the best alternative. On the surface, this solution may appear to be a good option because it collects millions of measurements that are otherwise unavailable. The reality is that big data is only relevant when you can build true value out of the entire data set and, while cell phones provide millions of measurements, the resulting cleaned data remains too inaccurate for crowd-sourced weather data to provide a reliable dataset.

The alternative crowd-sourced weather networks that use consumer weather stations to collect data also lead to huge problems in data quality. These weather stations lack any sort of placement control. They can be installed next to a tree, by air conditioning units or on the side of a house – all of which cause inaccurate readings that lead to more flawed output. And although these types of weather stations are able to collect data on rain and wind, none are able to collect data on hail – which causes millions of dollars in insurance claims each year.

The Case for an Empirical Weather Network

To resolve the insurance industry’s blind spot, the solution must contain highly accurate weather data that can be translated into actionable items. IoT has changed what is possible, and, with today’s technology, insurers should be able to know exactly where severe weather has occurred and the severity of damage at any given location. The answer lies in establishing a more cost-effective weather station, one that is controlled and not crowd-sourced. By establishing an extensive network of weather stations with controlled environments, the data accuracy can be improved tremendously. With improved data accuracy, algorithms can be reviewed and enhanced so insurers can garner actionable data to improve their storm response and recovery strategies.

Creating an extensive network of controlled weather stations is a major step toward fixing the insurance industry’s data blind spot, but there is one additional piece of data that is required. It is imperative that these weather stations measure everything, including one of the most problematic and costly weather events – hail. Without gathering hail data, the data gathered by the controlled weather stations would still be incomplete. No algorithm can make up for missing this entire category of data.

See also: 4 Benefits From Data Centralization

While technology has improved tremendously over the past 10 years, many insurers continue to use traditional data that has always been available them. Now is the time for insurers to embrace a new standard for weather data to gain insights that eliminate their blind spot, improve their business and provide better customer experiences.

Understory has deployed micro-networks of weather stations that produce the deep insights and accuracy that insurers need to be competitive today. Understory’s data tracks everything from rain to wind to temperature and even hail. Our weather stations go well beyond tracking the size of the hail; they also factor in the hail momentum, impact angle and size distribution over a roof. This data powers actionable insights