Tag Archives: Bing

Demystifying “The Dark Web”

We often hear reference to the “deep” or “dark” web. What exactly is the deep or dark web? Is it as illicit and scary as it is portrayed in the media?

This article will provide a brief overview and explanation of different parts of the web and will discuss why you just might want to go there.


The surface web or “Clearnet” is the part of the web that you are most familiar with. Information that passes through the surface web is not encrypted, and users’ movements can be tracked. The surface web is accessed by search engines like Google, Bing or Yahoo. These search engines rely on pages that contain links to find and identify content. Search engine companies were developed so that they can quickly index millions of web pages in a short time and to provide an easy way to find content on the web. However, because these search engines only search links, tons of content is being missed. For example, when a local newspaper publishes an article on its homepage, that article can likely be reached via a surface web search engine like Yahoo. However, days later when the article is no longer featured on the homepage, the article might be moved into the site’s archive format and, therefore, would not be reachable via the Yahoo search engine. The only way to reach the article would be through the search box on the local paper’s web page. At that time, the article has left the surface web and has entered the deep web. Let’s go there now…


The deep web is a subset of the Internet and is not indexed by the major search engines. Because the information is not indexed, you have to visit those web addresses directly and then search through their content. Deep web content can be found almost anytime you do a search directly in a website — for example, government databases and libraries contain huge amounts of deep web data. Why does the deep web exist? Simply because the Internet is too large for search engines to cover completely. Experts estimate that the deep web is 400 to 500 times the size of the surface web, accounting for more than 90% of the internet. Now let’s go deeper…


The dark web or “darknet” is a subset of the deep web. The dark web refers to any web page that has been concealed because it has no inbound links, and it cannot be found by users or search engines unless you know the exact address. The dark web is used when you want to control access to a site or need privacy, or often because you are doing something illegal. Virtual private networks (VPNs) are examples of dark web sites that are hidden from public access unless you know the web address and have the correct log-in credentials.

One of the most common ways to access the dark web is through the Tor network. The Tor network can only be accessed with a special web browser, called the Tor browser. Tor stands for “ The onion router” and is referred to as “Onionland.” This “onion routing” was developed in the mid-1990s by a mathematician and computer scientists at the U.S. Naval Research Laboratory with the purpose of protecting U.S. intelligence communications online. This routing encrypts web traffic in layers and bounces it through random computers around the world. Each “bounce” encrypts the data before passing the data on to its next hop in the network. This prevents even those who control one of those computers in the chain from matching the traffic’s origin with its destination. Each server only moves that data to another server, preserving the anonymity of the sender.

Because of the anonymity associated with the Tor network and dark web, this portion of the Internet is most widely known for its illicit activities, and that is why the dark web has such a bad reputation (you might recall the infamous dark web site, Silk Road, an online marketplace and drug bazaar on the dark web). It is true that on the dark web you can buy things such as guns, drugs, pharmaceuticals, child porn, credit cards, medical identities and copyrighted materials. You can hire hackers to steal competitors’ secrets, launch a DDOS (distributed denial of service) attack on a rival, or hack your ex-girlfriend’s Facebook account. However, the dark web accounts for only about .01% of the web.

Some would say that the dark web has a bad rap, as not everything on the dark web is quite so “dark,” nefarious or illegal. Some communities that reside on the dark web are simply pro-privacy or anti-establishment. They want to function anonymously, without oversight, judgment or censorship. There are many legitimate uses for the dark web. People operating within closed, totalitarian societies can use the dark web to communicate with the outside world. Individuals can use the dark web news sites to obtain uncensored new stories from around the world or to connect to sites blocked by their local Internet providers or surface search engines. Sites are used by human rights groups and journalists to share information that could otherwise be tracked. The dark net allows users to publish web sites without the fear that the location of the site will be revealed (think political dissidents). Individuals also use the dark web for socially sensitive communications, such as chat rooms and web forums for sensitive political or personal topics.


Don’t be afraid – dive deeper!

Download the Tor browser at www.torproject.org and access the deep/dark web information you have been missing. Everything you do in the browser goes through the Tor network and doesn’t need any setup or configuration from you. That said, because your data goes through several relays, it can be slow, so you might experience a more sluggish Internet than usual. However, preserving your privacy might be worth the wait. If you are sick of mobile apps that are tracking you and sharing your information with advertisers, storing your search history, or figuring out your interests to serve you targeted ads, give the Tor browser a try.

How Google Thinks About Insurance

For those of us wondering what Google plans to do in insurance — isn’t that all of us? — it’s worth looking at the company’s Project Sunroof. The project uses exceptionally sophisticated mapping data to determine which homeowners would most benefit from solar panels and, in the process, may provide some insight into how Google is approaching insurance.

To me, there are two key aspects of Project Sunroof. The first is that Google is taking a bottom-up approach that could inform a lot of decisions about insurance (while insurers traditionally go top-down). The second is that Google is being unusually smart about combining layers of information — some proprietary, some in the public domain; some new, some long-available — to produce what my frequent co-author Chunka Mui has, with a little help from me, labeled “emergent knowledge.” (“Big data” is the term commonly used, but data isn’t very interesting, while knowledge is. And the size of the database doesn’t matter. What matters is using developments in technology to look in the right places to find the right data to answer the right questions so that revelations emerge.)

Top-down vs. bottom-up

Insurers typically start with pools of risk. They’re getting much more sophisticated about subdividing those pools into ever smaller groups, but the thinking is still along the lines of “drivers without moving violations who travel 11,000 to 12,000 miles a year in generally suburban conditions.” Insurers will keep getting more and more specific and produce more and smaller pools but are still going from the top down.

Now look at Project Sunroof. Google is modeling the world in three dimensions and using that model to generate information house by house based on totally personalized criteria: on the square footage on the roof that would be available for solar panels, on the amount of cloud cover that is expected to obscure the sun above that house, on the effectiveness of the sunlight that will hit the roof (incorporating calculations based on temperature and on how the angle of the sun changes each day) and on any shade that would be cast on those panels from other structures. Although the article doesn’t say so, I assume Google calculates potential savings on solar based on the rates of each local utility. In any case, there are no pools in sight for Google — unless you want it to tell you about those in the backyards.

That same model of the world could be the basis for a house-by-house, car-by-car, person-by-person approach to insurance for Google. And, if this approach works, Google will gain the sort of information advantage that has proved to be almost impossible to overcome. Even the largest insurers would have a hard time spending the money that Google has to map the U.S. by having cars drive every single street to take pictures and collect data, by making a series of acquisitions of data providers and by employing a small army of people to manually fix errors and update maps — and Google would still have a years-long head start in developing its model of the world. Microsoft has thrown billions of dollars at search engines, but even Microsoft couldn’t overcome the fact that Google’s dominant share meant it was always learning and improving faster than Microsoft’s Bing. Apple’s map services were ridiculed, by comparison with Google’s, when Apple launched them in September 2012. Apple is now at least in Google’s ballpark on mapping, but no insurer can come close to Apple’s resources — a $646 billion market valuation and $202.8 billion of cash in the bank. That’s “billion,” with a “b.”

Emergent knowledge

Google obviously begins with a huge asset because of its prescient decision years ago to map the entire U.S. and because of the recent work that has made that map 3D.  But Google is also taking data wherever it can get it.

I know from some work I did at the Department of Energy in 2010 that national maps of sunlight have been available for years, and they have surely become far more detailed as the interest in solar power has spread, so I assume Google didn’t have to generate those maps on its own. Temperature maps are also in the public domain. (Especially high or low temperatures degrade the performance of solar panels.) Those maps will become increasingly granular as they incorporate data from smartphones and other widely used devices that can act as sensors — temperature will no longer be what the weather station reports from the Detroit airport; temperature will be known house by house. Overhead photos from satellites and, in some cases, drones are widely available, so Google can use those to check square footage of roofs, to see which direction the solar panels would point and so on. Google can collect information on rates from state utility commissions, where utilities have to make regular filings.

It’s easy to imagine Google layering similar types of information onto its map of the world for insurance purposes. In response to the federal Data.gov initiative, governments at all levels are making more information available digitally, so Google could incorporate lots of data about where and when accidents occur, where break-ins happen, where and when muggings occur and so on.

Google could incorporate private work that is taking a 3D approach to flood risk (whether your house is three feet higher or lower than the average in a neighborhood can make all the difference) and is being much more discriminating about earthquake risk. Google could add information, from public or private sources, on the age of homes, type of pipes used, appliances, etc. to flesh out its understanding of the risks in homes.

And, of course, Google will have lots of very precise information of its own to add to its model of the world, based on, for instance, what it knows about where your smartphone is and can infer about where you park your car, where and when you drive, etc.

Once you take all this information and map it to such a precise model, there will surely be some non-obvious and highly valuable insights.

WWGD: What Will Google Do?

Looking at Project Sunroof still doesn’t say a lot about how Google will attack insurance. Will it just sell increasingly targeted and valuable ads? Will it sell leads? Will it become a broker? Will it do more?

But I think it’s safe to say that, whatever Google does, its starting point will the most sophisticated model of the world — and that model will always be improving.

Restoring the Agent-Client Relationship

There has been a lot of frustration in the insurance industry from both those who sell it and those who need it. Both camps are suffering financially, and both can do better if they get together on vital insurance protection, but they just can’t seem to hook up without jumping through hoops. In an era of hyper-information and instant communication, this disconnect may seem crazy, but it’s real. In a time of chaotic change, an online meetup service would be valuable to agents and consumers alike to repair that agent/client relationship and put the personal touch back into insurance.

Financially challenged agents and consumers

Incomes have stagnated for insurance agents and the general public alike, and the route to better times seems unclear for both sides. According to the U.S. Bureau of Labor Statistics, the mean annual salary for insurance sales agents inched up only 2% between 2010 and 2014. In 2010, it was $62,520.  In 2014, it was $63,730. Furthermore, the field has become overcrowded, with 18% more agents vying for the business in 2014 than in 2010.

The general public has fared no better. Following the Great Recession, which began in December 2007, the wealthiest Americans have done well. The “rest of us,” however, continue to struggle. The proportion of American households defined as “middle-income” remained stagnant from 2010 through 2014, at about 51%, according to a Pew Research Center study. Back in 1970, the “middle-income” percentage was 10 points higher.

The potential for pocketbook improvements

Both insurance agents and their prospects could do better financially if they could somehow get together more quickly and smoothly – through a matchmaker or intermediary.

It’s easy to see how agents could profit. With, say, a 10% to 50% increase in qualified leads per month, a corresponding jump in income could be expected. And with other efficiencies through more nuanced matchmaking, even greater income increases might be forthcoming – through enhanced referrals, for example.

It’s a little more complicated to see how connecting more smoothly could financially benefit insurance buyers. It becomes clear, though, when one goes to the heart of what insurance is for.  It’s for mitigating risk, which can be expensive. It also means personal benefits such as improved health and well-being. For example, good guidance from an agent can:

  • Make the difference between paying and not having to pay for home repairs after a type of storm damage not covered by an “economy” policy the agent advised against.
  • Preserve a family’s estate by convincing the family, early on, of the prudence of securing long-term-care insurance. This could be a financial game changer for millions of families that are now exposed. According to industry estimates, about 90% of those who could benefit from LTC insurance do not own a policy. And one of the biggest causes of bankruptcy is uncovered health expenses, especially in the later years!
  • Help keep clients safe and whole through an auto policy with safe-driving incentives. The potential benefits range from lower premiums to higher lifetime incomes because of avoiding accidents that might interrupt the ability to work.

As technology and society evolve, good guidance from an insurance agent may affect people’s pocketbooks and lives in more significant ways than ever. More and more agents can:

  • Team with financial advisers to foster sound budgeting, savings, investments and money management.
  • Influence their clients’ health by recommending policies, now starting to appear, that come with fitness incentives. The financial win here is double: lower premiums for keeping up one’s wellness routine and greater lifetime earnings through enhanced vitality and work-span.

The matchmaker solution

A good matchmaking service brings insurance agents and buyers together in very efficient, human ways. It starts with search and ends with introductions and contact.  It includes:

  • A search function to locate agents for a particular type of insurance (auto, critical illness, health, homeowners, life, long-term care, Medicare supplement) in a particular geographic area.
  • A list of agents with their pictures and names visible, for the buyer to peruse and select from.
  • Details about each agent, including:
    • Insurance lines and carriers represented.
    • Agent’s biography or background description.
    • Reviews or testimonials with ratings (usually one to five stars).
    • Link to the agent’s personal or business website.
    • Other information ranging from a location map to social media links.

Limited matching has existed for a few years. Some generic consumer rating and matching services embrace insurance agents. They include Yelp and Angie’s List. General search services, such as Google and Bing, serve as de facto matching services, but in a very spotty way. Insurance associations develop leads that are sold to agents but do not typically provide free online access to individual agents.

Robust agent-buyer matching, with all the above elements, is ready for prime time. In 2015, Agent Review, the first complete rating and matching service designed specifically for insurance agents and insurance buyers, was introduced.