The Safety of Internet Search Engines - Revisited

December 11 , 2006

Abstract

In May 2006 McAfee released a study comparing the safety of leading search engines. This report updates that study with new data and deeper analysis of the specific types of threats posed by search engine results.

In this study, we compare the safety of leading search engines, using McAfee SiteAdvisor's automated Web site ratings. We again find that most leading search engines are similar in the safety of the sites they link to, though AOL replaces MSN as the safest engine and Yahoo! replaces Ask as the engine with the most risky results. Across search engines, we find sponsored results significantly less safe than search engines' organic results. Unsavory e-mail conduct is the dominant security risk although search engine users are also heavily exposed to risky downloads, browser exploits, and scams.

% of Red1 and Yellow2 Sites...

By Search Engine
% of Red and Yellow Sites - by Search Engine

By Type of Result
% of Red and Yellow Sites - by Result Type

Key Findings
  • In the U.S., the top 5 search engines (representing 91% of all search engine use) all include some dangerous results.

  • AOL returns the safest results with 3.6% of results rated red1 or yellow2 by McAfee SiteAdvisor. At 5.1%, Yahoo! returns the most results rated red or yellow. On average, 4.4% of search results link to risky Web sites.

  • 8% of sponsored results are rated red or yellow - almost three times the percentage of red and yellow sites found in organic results. Notably, scam sites are found at a much greater frequency in sponsored results.

  • 41.4% of red or yellow sites exhibit poor e-mail practices, 24.5% contain risky downloads, 26.8% are scam sites, 32.3% link to other risky sites and 3.0% contain browser exploits. Many sites posed multiple dangers.

  • Adult search terms are twice as dangerous as non-adult search terms.

  • Dangerous sites exceeded 90% of results for certain risky keywords. Particularly dangerous keywords include "bearshare", "free screensavers", and "free ringtones".

 

1 - "Red" rated sites failed SiteAdvisor's safety tests. Examples are sites that distribute adware, send a high volume of spam, or make unauthorized changes to a user's computer.

2 - "Yellow" rated sites engage in practices that warrant important advisory information based on SiteAdvisor's safety tests. Examples are sites which send a high volume of "non-spammy" email, display many popup ads, or prompt a user to change browser settings.

Introduction

Search engines have become users' primary gateway to the Internet. Nearly 80% of Web site page visits originate from search engines. Although search engines often provide the fastest way of navigating the Web, they are not necessarily the safest.

In general, search engine rankings do not explicitly focus on Web site security. But users often make assumptions based on sites' search rankings. Over one third of searchers perceive that companies ranked high in search results are among the top companies in their field, indicating that users may falsely associate trustworthiness with search result ranking. But search rankings are not a reflection of safety. Unless users are selecting familiar Web sites that they trust from previous experience, users risk exposing themselves to numerous threats.

Methodology

We compare the safety of search results from five search engines: Google, Yahoo!, MSN, AOL, and Ask. First, we compiled a list of approximately 2,500 popular keywords derived from lists of common searches from Google Zeitgeist, Yahoo!, AOL, Ask, Lycos, Wordtracker, Hitwise and other industry sources. We assess the first five pages of search results for each keyword on each of the five search engines. We analyze site safety based on result position and result type (organic versus sponsored).

Our site safety assessments come from McAfee SiteAdvisor's Web safety database of 7.9 million of the most trafficked Web sites. Since our earlier search engine analysis in May 2006, McAfee SiteAdvisor has increased its coverage of Web sites, site downloads and browser exploits -- growing from 3 million sites to 7.9 million.

We analyze Web site safety using overall McAfee SiteAdvisor ratings as well as component ratings of specific behaviors: browser exploits, e-mail, downloads, scams, annoyances (such as pop-ups), and links to other such sites.

McAfee SiteAdvisor Rating Sources

Analysis
Search Engine Comparison

While all search engines return some unsafe results, listings at Yahoo! are particularly risky. In our analysis, 5.1% of Yahoo! results are rated red or yellow. AOL returns the safest results with 3.6% rated red or yellow. Overall, 4.4% of search results are rated red (2.6%) or yellow (1.7%).

Percent of red and yellow results by search engine
Percentage of red and yellow results by search engine

Since the release of our search engine study in May 2006, the percentage of red and yellow sites in search results decreased from 5.0% to 4. 4%. Google, AOL, and Ask now return safer results, while Yahoo! and MSN return riskier results. Several factors may contribute to these changes. For example, some search engines have changed their ranking algorithms. Other search engines have changed ad platforms altogether. At the same time, we have updated our keyword list since our prior analysis, in an effort to better reflect the newest and most current top search terms.

The relative rankings of the search engines also shifted. AOL replaces MSN as the search engine returning the safest results, and Yahoo! replaces Ask as the search engine returning the most risky results. MSN's safety decline is likely due to the expansion of its paid search program, which was in transition (showing unusually few ads) during data collection for our prior article. The percentage of red and yellow sites in MSN's sponsored results jumped from 6.3% in May to 10.7% in November.

  November 2006 May 2006
Google   () 4.2% 5.3%
Yahoo   () 5.1% 4.3%
MSN   () 4.6% 3.9%
AOL   () 3.6% 5.3%
Ask    () 4.2% 6.1%
     
Organic   () 3.0% 3.1%
    Sponsored   ()     8.0% 8.5%

safer since last study            less safe since last study

Results by Safety Facet: Exploits, E-mail, Downloads, Scams, Links

Search engine users are exposed to several distinct types of Web site security risks. Of those search results rated red or yellow, 24.5% received those ratings due to risky downloads, 41.4% for poor e-mail practices, 26.8% for scams, and 31.2% for links to other sites with such practices. 3.0% of red and yellow search results contain browser exploits -- particularly serious threats which can damage a user's PC as a user merely browses a site. Google's recent implementation of an interstitial warning page for certain exploit sites slightly improves the safety of Google's exploit results. This recent development represents an effort by Google to steer users away from dangerous sites, but we find Google warnings for only 18% of Google search results containing browser exploits, so users cannot rely on these warnings to stay safe.

% Search Results Rated Red/Yellow By Security Risk
    Downloads   E-mail Exploits Scams Links
Google 1.1% 1.9% 0.13% 0.9% 1.6%
Yahoo! 1.0% 2.0% 0.09% 2.1% 1.6%
MSN 1.3% 1.6% 0.09% 1.21% 1.3%
AOL 1.1% 2.0% 0.17% 0.5% 1.2%
Ask 1.0% 1.6% 0.17% 1.2% 1.4%
Overall 1.1% 1.8% 0.13% 1.2% 1.4%

 

% Red/Yellow Search Results By Security Risk
    Downloads   E-mail Exploits Scams Links
Google 26.2% 45.2% 3.1% 21.4% 38.1%
Yahoo! 19.6% 39.2% 1.8% 41.2% 31.4%
MSN 28.3% 34.8% 2.0% 26.1% 28.3%
AOL 30.6% 55.6% 4.7% 13.9% 33.3%
Ask 23.8% 38.1% 4.0% 28.6% 33.3%
Overall 25.0% 41.4% 3.0% 26.8% 32.3%

 

These tables confirm that users face differing risks at the various search engines. Yahoo's search results include twice as many scam results as others, while AOL's search results contain the fewest scam sites. (SiteAdvisor scam ratings warn users about misleading site content such as deceptive claims, offers or billing practices.) But AOL isn't safer across the board: Exploits and bulk e-mailers are more widespread among AOL-listed results.

Percent of red and yellow scam results by search engine
Percentage of red and yellow scam results by search engine

Organic vs. Sponsored Results

In our analysis, search engines' sponsored (paid advertising) results are approximately three times as likely to lead to red and yellow sites as are organic (non-paid) results. This result reflects well on organic search ranking algorithms, but it also indicates that search engines receive substantial payments from risky sites. By allowing risky Web sites to buy prominent placement within search results, search engines help these sites reach -- and, potentially, harm -- unsuspecting users. Search advertising will exceed $16 billion in 2006. Assuming unsafe sites' ad prices are similar to other advertisers (we think, a reasonable approximation), search engines earn approximately $1.28 billion annually from their U.S. operations by sending users to risky sites.

Red and yellow sites appear in sponsored results at almost three times the rate of organic results.
Red and yellow sites appear in sponsored results at almost three times the rate of organic results.

MSN has the highest percentage of risky sponsored results (10.7%), while Ask's sponsored results are safest (6.5%).

Percentage of red and yellow sponsored results by search engine.
Percentage of red and yellow sponsored results by search engine.

Sponsored Results Promoting Scam Sites

Most notably, sponsored results contain a larger percentage of scam sites than organic results. 4.1% of sponsored results are rated red or yellow for scams as compared to 0.1% of organic results. Scam sites contain misleading site content such as deceptive claims or promotional offers and bait-and-switch billing practices. For example, scam sites may attach fees to free downloads, charge users to enter the free Green Card Lottery, or disguise pyramid schemes as lucrative work-at-home opportunities. Scam sites appear frequently in sponsored results despite search engines' advertising guidelines prohibiting deceptive content. See also Ben Edelman 's False and Deceptive Pay-Per-Click Ads.

Red and yellow scam results appear almost entirely in sponsored listings.
Red and yellow scam results appear almost entirely in sponsored listings.

The Use of the Word "Free"

Popular searches containing the word "free" are particularly likely to lead users to sites with unsavory practices. For popular searches containing the word "free," 14.5% of search results link to sites rated yellow or red. Web sites buying the paid search word "Free" use a variety of strategies to ensnare users. "Free" downloads often come bundled with unwanted programs, especially adware, toolbars, or tracking software. Sites offering "free" services (like ringtones or credit checks) often make misleading claims, bury fee disclosures in inconspicuous fine print, or impose other deceptive billing practices. Other sites prey on naive consumers by charging for products that users could easily get elsewhere for free. For example, some sites charge users for the free browser Firefox, justifying the fee with dubious promises of technical support or customer service. Finally, sites claiming to offer free products (like iPods or flat screen TVs) often share users' e-mail addresses with third parties -- burdening affected users with hundreds of e-mails per week.

Percentage of red and yellow results for search terms containing the word 'free.'
Percentage of red and yellow results for search terms containing the word "free."

Safety by Page

Search result safety remains roughly constant across the first five pages of results, indicating a lack of correlation between search result ranking and site safety. First page results, which are selected by 62% of searchers, are no safer than results on subsequent pages. So users can't keep themselves safe simply by staying on top-rated search results.

Percentage of red and yellow results by search result page.
Percentage of red and yellow results by search result page.

Safety by Position

Organic search result safety is consistent across position rankings, but sponsored search result safety varies by ranking. In our tests, Google's first sponsored result on the top of the page is somewhat more likely to be safe, relative to Google's positions 2 and 3. Among Google sponsored results listed on the right side of the page, the top two results are less risky than subsequent positions. We've noticed that many #1-rated Google ads are actually non-profits receiving free advertising through Google.org. The special characteristics of these non-profits probably explain the relative safety of Google's top-most position: These ads tend to promote sites that are safe (no spyware, scams, or the like), and these ads also tend to be top-ranked since their creators need not make cash payments to Google.

Organic
Position % Red/Yellow
1 3.1%
2 3.2%
3 3.5%
4 2.9%
5 3.2%
6 2.9%
7 3.0%
8 3.1%
9 3.1%
10 3.2%
 
Sponsored: Top of page
Position % Red/Yellow
1 6.2%
2 9.5%
3 9.0%
 
Sponsored: Right side of page
Position % Red/Yellow
1 5.2%
2 6.2%
3 8.4%
4 9.1%
5 8.9%
6 8.3%
7 7.4%
8 7.7%

Google's Results by Rank

Category Analysis

Of the Google Zeitgeist search terms we analyze, the most dangerous Zeitgeist category is "tech toys" -- where 23.3% of results are rated red or yellow by McAfee SiteAdvisor. "Tech toys" search terms include "iPod nano", "mp3 music downloads", and "winmx" -- terms that often lead to scams and other dubious practices. Overall, music and technology related Google Zeitgeist categories return the greatest percentages of risky results. Security hazards for these categories primarily include risky downloads and scams charging for free software.

The top five most dangerous Google Zeitgeist categories include:

  Google Zeitgeist Category     % Red/Yellow Results  
Tech Toys 23.3%
Popular Software 20.0%
Digital Music 19.7%
Technology Queries 18.6%
Popular Male Singers 15.3%

Celebrity searches frequently contain many risky sites. We found many dangerous sites in searches for "popular male singers" (15.3%) such as "Usher" and "Nelly," as well as "popular women" (10.5%) such as "Lindsay Lohan" and "Paris Hilton". Dangerous search results were also widespread in "top sports queries" (11.3%) such as "Real Madrid" and "cricket," as well as "childhood favorites" (6.7%) which included keywords such as "Winnie the Pooh" and "Tweety".

In contrast, searches for keywords in the Google Zeitgeist category "Web 2.0" such as "wikipedia" and "youtube" do not contain any dangerous search results. Among other relatively safe Google Zeitgeist categories are "summer travel" (0.8%), which includes search terms such as "Expedia" and "Travelocity", and "weekend errands" (0.4%), which includes "USPS" and "Home Depot".

Individual Keyword Analysis

In our tests, the single most dangerous keyword across all search engines is "bearshare" where 53.3% of search results are rated red or yellow. Bearshare is a free file sharing program that, in our tests, bundled adware during its installation. "Bearshare" is also a risky keyword because many sites charge users to access this free program. The percentage of red and yellow search results peaked at 90.9% in a search for "rotten.com" on MSN.

The top ten most dangerous search terms:

  Search Term   % Red/Yellow Results  
1 bearshare 53.3%
2 rotten.com 53.1%
3 free screensavers 53.0%
4 winmx 52.0%
5 screensavers 48.2%
6 limewire 46.2%
7 kazaa 45.8%
8 free ringtones 44.3%
9 ringtones 43.8%
10 lime wire 42.7%

 

Adult vs. Non-adult Search Terms

Our tests suggest that adult search terms are twice as likely to lead to unsafe results as are non-adult search terms. To investigate the safety of adult search terms, we formed a set of 142 top adult keywords. 8.0% of results for these adult search terms are rated red or yellow, versus 4.1% for non-adult terms. Adult keywords returned a higher percentage of sites containing browser exploits and sites with poor e-mail practices.

Adult search terms are twice as likely to lead to dangerous search results as non-adult search terms.
Adult search terms are twice as likely to lead to dangerous search results as non-adult search terms.

Discussion

It remains to be seen whether search engines' rankings will eventually evolve to incorporate site safety. Google's ad rankings already incorporate a notion of "landing page quality" -- and Google's ad guidelines mention several factors that ring true to us, like avoiding sending users excessive e-mails, and avoiding making users escape "excessive obstacles" before getting a promised free product. Still, we're alarmed that so many Google advertisers continue to promote services that, to us, seem like outright scams -- promising something is "free" when it's not, or charging for something that is widely available elsewhere for free. More generally, search engines make profits from scammers' sponsored results -- a factor that tends to discourage search engines from aggressively removing unsafe sites from their results. Rigid advertising guidelines and tough filtering might reduce search engine revenue in the short run, but such practices could ultimately benefit search engines. If sponsored results became safer than organic results, users might click sponsored links more frequently, and advertisers might increase their paid search spending accordingly.

The ease of Web anonymity creates a special problem that search engines are well-equipped to solve. If search engines look the other way, scammers can manipulate the market, trick consumers, and, it seems, largely get away with it. But search engines have an opportunity to help users avoid dangerous sites by limiting who can become an advertiser and by prohibiting deceptive and malicious ads. Meanwhile, interested users can get many of these benefits by installing the McAfee SiteAdvisor browser plug-in, which automatically annotates search results to report the McAfee SiteAdvisor safety assessment of each search result.

Some users remain naïve about Web safety. They click, submit e-mail addresses, and download freely without realizing the consequences until the damage has been done. As Web safety proponents we take that to heart, since that means their online experience is ultimately worse than in a safer world. Other users seek to avoid the dangers of the Web by limiting their activities online: They only visit familiar sites, never give out their e-mail addresses, and never download new programs. That's unfortunate also: Their caution prevents them from exploring the full Web and from benefiting from many useful and entertaining sites. Users should be able to safely navigate the Web without undue concern. We're hopeful that search engines will take steps to improve the safety of their results, to make it that much easier for users to do so.

 

Methodology 

To generate an overall snapshot of the risks faced by users in different countries, we also examine top searches performed at country-specific search engines outside the U.S.  In contrast to our U.S. analysis which uses the same 2,500 keywords across all search engines, our international analysis uses each country’s most popular 15 or so keywords in their local language.  This ensures we are appropriately incorporating differences in local consumer searching patterns rather than artificially mandating a fixed keyword list to test across all countries.  For data on country-specific top searches, we turn to Google Zeitgeist

At the same time, we recognize that users in different countries tend to use country-specific search engines, rather than simply (say) google.com.  So our international analysis uses results from searches run on each country’s corresponding regional Google site.  For international analysis, we analyze the first page of regional keyword search results. 

Results 

Popular search terms vary across countries, and regional sites (e.g. xyz.de) are ranked more favorably on their respective regional search engines (e.g. google.de) than on general search engines.  Comparing the safety of the most popular searches conducted on each of 30 Google regional domains, we find that the most dangerous searches are being conducted on Google’s Indian regional domain, google.co.in. 9% of results for India’s most popular search terms are rated red or yellow by McAfee SiteAdvisor.  Among India’s most popular risky keywords are Indian actors Salman Khan and Aishwarya Rai.  Relatively risky searches are also being conducted by searchers in Greece (7.4%), where popular search terms include the television series “Rebelde Way” and pop band “Erreway”.  Searchers in Finland and Ireland are performing the safest searches: only 0.7% and 0.9% of results for the most popular searches on google.fi and google.ie are rated red or yellow. 

Discussion 

Search safety varies for search engine users in different countries.  Popular culture and consumer trends vary across the world, and in turn, opportunities for consumer exploitation vary as well.  Advertisers bidding on popular country-specific keywords can target specific audiences by selecting language and country preferences.  Therefore relatively risky sponsored results for regional search terms may indicate greater malicious activity within the given region. In addition, regional sites gain greater visibility within regional search results, so searchers face greater risks where there is a relatively high percentage of dangerous regional domains.  But even though searchers in some regions may be somewhat safer than searchers in another region, scammers and bad actors are equipped to reach consumers in every corner of the world. 

Search safety varies across regional search engines.
Search safety varies across regional search engines.

 

Analysis of Organic Results by Search Engine and Analysis of Sponsored Results by Search Engine

Analysis by Zeitgeist Keyword Group (safest and riskiest keyword categories)

Analysis by Individual Keyword

Resources

Forrester to Marketers: Let Age Define Your Buy - ClickZ

False and Deceptive Pay-Per-Click Ads - Ben Edelman

Google Revenue to Exceed $10 Billion in 2007 - eMarketer

How Consumers Find Web Sites In 2006 - Forrester

The Philanthropic Arm of Google - Google

Search Engine User Behavior Study - iProspect

Search Engine Market Share - Nielsen/NetRatings

Google AdWords Landing Page Quality Ranking Initiated - Search Engine Journal

 

Search Engine Advertisement Guidelines

Google - AdWords Editorial Guidelines

Google AdWords Landing Page and Site Quality Guidelines

Inside AdWords: Landing Page Quality Update

Yahoo! - Sponsored Search Listing Guidelines

MSN - AdCenter Search Ads Content Guidelines

AOL - Ad Specs - Policies and Guidelines

Ask - Sponsored Listings - Guidelines

 

The SiteAdvisor Web Safety Tool

siteadvisor.com