The Safety of Internet Search Engines - RevisitedDecember 11 , 2006 |
||
|
Ben Edelman |
Hannah Rosenbaum |
|
Abstract
In May 2006 McAfee released a study comparing the safety of leading search engines. This report updates that study with new data and deeper analysis of the specific types of threats posed by search engine results. In this study, we compare the safety of leading search engines, using McAfee SiteAdvisor's automated Web site ratings. We again find that most leading search engines are similar in the safety of the sites they link to, though AOL replaces MSN as the safest engine and Yahoo! replaces Ask as the engine with the most risky results. Across search engines, we find sponsored results significantly less safe than search engines' organic results. Unsavory e-mail conduct is the dominant security risk although search engine users are also heavily exposed to risky downloads, browser exploits, and scams. |
||
% of Red1 and Yellow2 Sites...
By Search Engine
By Type of Result
-
In the U.S., the top 5 search engines (representing 91% of all search engine use) all include some dangerous results.
-
AOL returns the safest results with 3.6% of results rated red1 or yellow2 by McAfee SiteAdvisor. At 5.1%, Yahoo! returns the most results rated red or yellow. On average, 4.4% of search results link to risky Web sites.
-
8% of sponsored results are rated red or yellow - almost three times the percentage of red and yellow sites found in organic results. Notably, scam sites are found at a much greater frequency in sponsored results.
-
41.4% of red or yellow sites exhibit poor e-mail practices, 24.5% contain risky downloads, 26.8% are scam sites, 32.3% link to other risky sites and 3.0% contain browser exploits. Many sites posed multiple dangers.
-
Adult search terms are twice as dangerous as non-adult search terms.
-
Dangerous sites exceeded 90% of results for certain risky keywords. Particularly dangerous keywords include "bearshare", "free screensavers", and "free ringtones".
1 - "Red" rated sites failed SiteAdvisor's safety tests. Examples are sites that distribute adware, send a high volume of spam, or make unauthorized changes to a user's computer.
2 - "Yellow" rated sites engage in practices that warrant important advisory information based on SiteAdvisor's safety tests. Examples are sites which send a high volume of "non-spammy" email, display many popup ads, or prompt a user to change browser settings.
Search engines have become users' primary gateway to the Internet. Nearly 80% of Web site page visits originate from search engines. Although search engines often provide the fastest way of navigating the Web, they are not necessarily the safest.
In general, search engine rankings do not explicitly focus on Web site security. But users often make assumptions based on sites' search rankings. Over one third of searchers perceive that companies ranked high in search results are among the top companies in their field, indicating that users may falsely associate trustworthiness with search result ranking. But search rankings are not a reflection of safety. Unless users are selecting familiar Web sites that they trust from previous experience, users risk exposing themselves to numerous threats.
We compare the safety of search results from five search engines: Google, Yahoo!, MSN, AOL, and Ask. First, we compiled a list of approximately 2,500 popular keywords derived from lists of common searches from Google Zeitgeist, Yahoo!, AOL, Ask, Lycos, Wordtracker, Hitwise and other industry sources. We assess the first five pages of search results for each keyword on each of the five search engines. We analyze site safety based on result position and result type (organic versus sponsored).
Our site safety assessments come from McAfee SiteAdvisor's Web safety database of 7.9 million of the most trafficked Web sites. Since our earlier search engine analysis in May 2006, McAfee SiteAdvisor has increased its coverage of Web sites, site downloads and browser exploits -- growing from 3 million sites to 7.9 million.
We analyze Web site safety using overall McAfee SiteAdvisor ratings as well as component ratings of specific behaviors: browser exploits, e-mail, downloads, scams, annoyances (such as pop-ups), and links to other such sites.
While all search engines return some unsafe results, listings at Yahoo! are particularly risky. In our analysis, 5.1% of Yahoo! results are rated red or yellow. AOL returns the safest results with 3.6% rated red or yellow. Overall, 4.4% of search results are rated red (2.6%) or yellow (1.7%).
Percentage of red and yellow results by search engine
Since the release of our search engine study in May 2006, the percentage of red and yellow sites in search results decreased from 5.0% to 4. 4%. Google, AOL, and Ask now return safer results, while Yahoo! and MSN return riskier results. Several factors may contribute to these changes. For example, some search engines have changed their ranking algorithms. Other search engines have changed ad platforms altogether. At the same time, we have updated our keyword list since our prior analysis, in an effort to better reflect the newest and most current top search terms.
The relative rankings of the search engines also shifted. AOL replaces MSN as the search engine returning the safest results, and Yahoo! replaces Ask as the search engine returning the most risky results. MSN's safety decline is likely due to the expansion of its paid search program, which was in transition (showing unusually few ads) during data collection for our prior article. The percentage of red and yellow sites in MSN's sponsored results jumped from 6.3% in May to 10.7% in November.
| November 2006 | May 2006 | |
Google ( ) |
4.2% | 5.3% |
Yahoo ( ) |
5.1% | 4.3% |
| MSN () |
4.6% | 3.9% |
| AOL () |
3.6% | 5.3% |
| Ask () |
4.2% | 6.1% |
| Organic () |
3.0% | 3.1% |
| Sponsored () |
8.0% | 8.5% |
safer since last study less safe since last study
Search engine users are exposed to several distinct types of Web site security risks. Of those search results rated red or yellow, 24.5% received those ratings due to risky downloads, 41.4% for poor e-mail practices, 26.8% for scams, and 31.2% for links to other sites with such practices. 3.0% of red and yellow search results contain browser exploits -- particularly serious threats which can damage a user's PC as a user merely browses a site. Google's recent implementation of an interstitial warning page for certain exploit sites slightly improves the safety of Google's exploit results. This recent development represents an effort by Google to steer users away from dangerous sites, but we find Google warnings for only 18% of Google search results containing browser exploits, so users cannot rely on these warnings to stay safe.
| % Search Results Rated Red/Yellow By Security Risk | |||||
| Downloads | Exploits | Scams | Links | ||
| 1.1% | 1.9% | 0.13% | 0.9% | 1.6% | |
| Yahoo! | 1.0% | 2.0% | 0.09% | 2.1% | 1.6% |
| MSN | 1.3% | 1.6% | 0.09% | 1.21% | 1.3% |
| AOL | 1.1% | 2.0% | 0.17% | 0.5% | 1.2% |
| Ask | 1.0% | 1.6% | 0.17% | 1.2% | 1.4% |
| Overall | 1.1% | 1.8% | 0.13% | 1.2% | 1.4% |
| % Red/Yellow Search Results By Security Risk | |||||
| Downloads | Exploits | Scams | Links | ||
| 26.2% | 45.2% | 3.1% | 21.4% | 38.1% | |
| Yahoo! | 19.6% | 39.2% | 1.8% | 41.2% | 31.4% |
| MSN | 28.3% | 34.8% | 2.0% | 26.1% | 28.3% |
| AOL | 30.6% | 55.6% | 4.7% | 13.9% | 33.3% |
| Ask | 23.8% | 38.1% | 4.0% | 28.6% | 33.3% |
| Overall | 25.0% | 41.4% | 3.0% | 26.8% | 32.3% |
These tables confirm that users face differing risks at the various search engines. Yahoo's search results include twice as many scam results as others, while AOL's search results contain the fewest scam sites. (SiteAdvisor scam ratings warn users about misleading site content such as deceptive claims, offers or billing practices.) But AOL isn't safer across the board: Exploits and bulk e-mailers are more widespread among AOL-listed results.
Percentage of red and yellow scam results by search engine
In our analysis, search engines' sponsored (paid advertising) results are approximately three times as likely to lead to red and yellow sites as are organic (non-paid) results. This result reflects well on organic search ranking algorithms, but it also indicates that search engines receive substantial payments from risky sites. By allowing risky Web sites to buy prominent placement within search results, search engines help these sites reach -- and, potentially, harm -- unsuspecting users. Search advertising will exceed $16 billion in 2006. Assuming unsafe sites' ad prices are similar to other advertisers (we think, a reasonable approximation), search engines earn approximately $1.28 billion annually from their U.S. operations by sending users to risky sites.
Red and yellow sites appear in sponsored results at almost three times the rate of organic results.
MSN has the highest percentage of risky sponsored results (10.7%), while Ask's sponsored results are safest (6.5%).
Percentage of red and yellow sponsored results by search engine.
Most notably, sponsored results contain a larger percentage of scam sites than organic results. 4.1% of sponsored results are rated red or yellow for scams as compared to 0.1% of organic results. Scam sites contain misleading site content such as deceptive claims or promotional offers and bait-and-switch billing practices. For example, scam sites may attach fees to free downloads, charge users to enter the free Green Card Lottery, or disguise pyramid schemes as lucrative work-at-home opportunities. Scam sites appear frequently in sponsored results despite search engines' advertising guidelines prohibiting deceptive content. See also Ben Edelman 's False and Deceptive Pay-Per-Click Ads.
Red and yellow scam results appear almost entirely in sponsored listings.
Popular searches containing the word "free" are particularly likely to lead users to sites with unsavory practices. For popular searches containing the word "free," 14.5% of search results link to sites rated yellow or red. Web sites buying the paid search word "Free" use a variety of strategies to ensnare users. "Free" downloads often come bundled with unwanted programs, especially adware, toolbars, or tracking software. Sites offering "free" services (like ringtones or credit checks) often make misleading claims, bury fee disclosures in inconspicuous fine print, or impose other deceptive billing practices. Other sites prey on naive consumers by charging for products that users could easily get elsewhere for free. For example, some sites charge users for the free browser Firefox, justifying the fee with dubious promises of technical support or customer service. Finally, sites claiming to offer free products (like iPods or flat screen TVs) often share users' e-mail addresses with third parties -- burdening affected users with hundreds of e-mails per week.
Percentage of red and yellow results for search terms containing the word "free."
Search result safety remains roughly constant across the first five pages of results, indicating a lack of correlation between search result ranking and site safety. First page results, which are selected by 62% of searchers, are no safer than results on subsequent pages. So users can't keep themselves safe simply by staying on top-rated search results.
Percentage of red and yellow results by search result page.
Organic search result safety is consistent across position rankings, but sponsored search result safety varies by ranking. In our tests, Google's first sponsored result on the top of the page is somewhat more likely to be safe, relative to Google's positions 2 and 3. Among Google sponsored results listed on the right side of the page, the top two results are less risky than subsequent positions. We've noticed that many #1-rated Google ads are actually non-profits receiving free advertising through Google.org. The special characteristics of these non-profits probably explain the relative safety of Google's top-most position: These ads tend to promote sites that are safe (no spyware, scams, or the like), and these ads also tend to be top-ranked since their creators need not make cash payments to Google.
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Google's Results by Rank
Of the Google Zeitgeist search terms we analyze, the most dangerous Zeitgeist category is "tech toys" -- where 23.3% of results are rated red or yellow by McAfee SiteAdvisor. "Tech toys" search terms include "iPod nano", "mp3 music downloads", and "winmx" -- terms that often lead to scams and other dubious practices. Overall, music and technology related Google Zeitgeist categories return the greatest percentages of risky results. Security hazards for these categories primarily include risky downloads and scams charging for free software.
The top five most dangerous Google Zeitgeist categories include:
| Google Zeitgeist Category | % Red/Yellow Results |
| Tech Toys | 23.3% |
| Popular Software | 20.0% |
| Digital Music | 19.7% |
| Technology Queries | 18.6% |
| Popular Male Singers | 15.3% |
Celebrity searches frequently contain many risky sites. We found many dangerous sites in searches for "popular male singers" (15.3%) such as "Usher" and "Nelly," as well as "popular women" (10.5%) such as "Lindsay Lohan" and "Paris Hilton". Dangerous search results were also widespread in "top sports queries" (11.3%) such as "Real Madrid" and "cricket," as well as "childhood favorites" (6.7%) which included keywords such as "Winnie the Pooh" and "Tweety".
In contrast, searches for keywords in the Google Zeitgeist category "Web 2.0" such as "wikipedia" and "youtube" do not contain any dangerous search results. Among other relatively safe Google Zeitgeist categories are "summer travel" (0.8%), which includes search terms such as "Expedia" and "Travelocity", and "weekend errands" (0.4%), which includes "USPS" and "Home Depot".
In our tests, the single most dangerous keyword across all search engines is "bearshare" where 53.3% of search results are rated red or yellow. Bearshare is a free file sharing program that, in our tests, bundled adware during its installation. "Bearshare" is also a risky keyword because many sites charge users to access this free program. The percentage of red and yellow search results peaked at 90.9% in a search for "rotten.com" on MSN.
The top ten most dangerous search terms:
| Search Term | % Red/Yellow Results | |
| 1 | bearshare | 53.3% |
| 2 | rotten.com | 53.1% |
| 3 | free screensavers | 53.0% |
| 4 | winmx | 52.0% |
| 5 | screensavers | 48.2% |
| 6 | limewire | 46.2% |
| 7 | kazaa | 45.8% |
| 8 | free ringtones | 44.3% |
| 9 | ringtones | 43.8% |
| 10 | lime wire | 42.7% |
Our tests suggest that adult search terms are twice as likely to lead to unsafe results as are non-adult search terms. To investigate the safety of adult search terms, we formed a set of 142 top adult keywords. 8.0% of results for these adult search terms are rated red or yellow, versus 4.1% for non-adult terms. Adult keywords returned a higher percentage of sites containing browser exploits and sites with poor e-mail practices.
Adult search terms are twice as likely to lead to dangerous search results as non-adult search terms.
It remains to be seen whether search engines' rankings will eventually evolve to incorporate site safety. Google's ad rankings already incorporate a notion of "landing page quality" -- and Google's ad guidelines mention several factors that ring true to us, like avoiding sending users excessive e-mails, and avoiding making users escape "excessive obstacles" before getting a promised free product. Still, we're alarmed that so many Google advertisers continue to promote services that, to us, seem like outright scams -- promising something is "free" when it's not, or charging for something that is widely available elsewhere for free. More generally, search engines make profits from scammers' sponsored results -- a factor that tends to discourage search engines from aggressively removing unsafe sites from their results. Rigid advertising guidelines and tough filtering might reduce search engine revenue in the short run, but such practices could ultimately benefit search engines. If sponsored results became safer than organic results, users might click sponsored links more frequently, and advertisers might increase their paid search spending accordingly.
The ease of Web anonymity creates a special problem that search engines are well-equipped to solve. If search engines look the other way, scammers can manipulate the market, trick consumers, and, it seems, largely get away with it. But search engines have an opportunity to help users avoid dangerous sites by limiting who can become an advertiser and by prohibiting deceptive and malicious ads. Meanwhile, interested users can get many of these benefits by installing the McAfee SiteAdvisor browser plug-in, which automatically annotates search results to report the McAfee SiteAdvisor safety assessment of each search result.
Some users remain naïve about Web safety. They click, submit e-mail addresses, and download freely without realizing the consequences until the damage has been done. As Web safety proponents we take that to heart, since that means their online experience is ultimately worse than in a safer world. Other users seek to avoid the dangers of the Web by limiting their activities online: They only visit familiar sites, never give out their e-mail addresses, and never download new programs. That's unfortunate also: Their caution prevents them from exploring the full Web and from benefiting from many useful and entertaining sites. Users should be able to safely navigate the Web without undue concern. We're hopeful that search engines will take steps to improve the safety of their results, to make it that much easier for users to do so.
Methodology
To generate an overall snapshot
of the risks faced by users in different countries, we also examine
top searches performed at country-specific search engines outside the
U.S. In contrast to our U.S. analysis which uses the same 2,500
keywords across all search engines, our international analysis uses
each country’s most popular 15 or so keywords in their local language.
This ensures we are appropriately incorporating differences in local
consumer searching patterns rather than artificially mandating a fixed
keyword list to test across all countries. For data on country-specific
top searches, we turn to
Google Zeitgeist.
At the same time, we recognize
that users in different countries tend to use country-specific search
engines, rather than simply (say)
google.com.
So our international
analysis uses results from searches run on each country’s corresponding
regional Google site. For international analysis, we analyze the
first page of regional keyword search results.
Results
Popular search terms vary across
countries, and regional sites
(e.g. xyz.de)
are ranked more favorably
on their respective regional search engines
(e.g. google.de)
than on general search engines. Comparing the safety of the most popular
searches conducted on each of 30 Google regional domains, we find that
the most dangerous searches are being conducted on Google’s Indian
regional domain,
google.co.in.
9% of results for India’s most popular
search terms are rated red or yellow by McAfee SiteAdvisor. Among
India’s most popular risky keywords are Indian actors Salman Khan
and Aishwarya Rai. Relatively risky searches are also being conducted
by searchers in Greece (7.4%), where popular search terms include the
television series “Rebelde Way” and pop band “Erreway”.
Searchers in Finland and Ireland are performing the safest searches:
only 0.7% and 0.9% of results for the most popular searches on
google.fi
and google.ie
are rated red or yellow.
Discussion
Search safety varies for search
engine users in different countries. Popular culture and consumer
trends vary across the world, and in turn, opportunities for consumer
exploitation vary as well. Advertisers bidding on popular country-specific
keywords can target specific audiences by selecting language and country
preferences. Therefore relatively risky sponsored results for
regional search terms may indicate greater malicious activity within
the given region. In addition, regional sites gain greater visibility
within regional search results, so searchers face greater risks where
there is a relatively high percentage of dangerous regional domains.
But even though searchers in some regions may be somewhat safer than
searchers in another region, scammers and bad actors are equipped to
reach consumers in every corner of the world.
Search safety varies across regional search engines.
Analysis of Organic Results by Search Engine and Analysis of Sponsored Results by Search Engine
Analysis by Zeitgeist Keyword Group (safest and riskiest keyword categories)
Analysis by Individual Keyword
Forrester to Marketers: Let Age Define Your Buy - ClickZ
False and Deceptive Pay-Per-Click Ads - Ben Edelman
Google Revenue to Exceed $10 Billion in 2007 - eMarketer
How Consumers Find Web Sites In 2006 - Forrester
The Philanthropic Arm of Google - Google
Search Engine User Behavior Study - iProspect
Search Engine Market Share - Nielsen/NetRatings
Google AdWords Landing Page Quality Ranking Initiated - Search Engine Journal
Search Engine Advertisement Guidelines
Google - AdWords Editorial Guidelines
Google AdWords Landing Page and Site Quality Guidelines
Inside AdWords: Landing Page Quality Update
Yahoo! - Sponsored Search Listing Guidelines
MSN - AdCenter Search Ads Content Guidelines
AOL - Ad Specs - Policies and Guidelines
Ask - Sponsored Listings - Guidelines
The SiteAdvisor Web Safety Tool
