Measuring personalization of web search

Abstract

Web search is an integral part of our daily lives. Recently, there has been a trend of personalization in Web search, where di erent users receive di erent results for the same search query. The increasing level of personalization is leading to concerns about Filter Bubble e ects, where certain users are simply unable to access information that the search engines' algorithm decides is irrelevant. Despite these concerns, there has been little quanti cation of the extent of personalization in Web search today, or the user attributes that cause it.In light of this situation, we make three contributions. First, we develop a methodology for measuring personalization in Web search results. While conceptually simple, there are numerous details that our methodology must handle in order to accurately attribute di erences in search results to personalization. Second, we apply our methodology to 200 users on Google Web Search and 100 users on Bing. We nd that, on average, 11.7% of results show di erences due to personalization on Google, while 15.8% of results are personalized on Bing, but that this varies widely by search query and by result ranking. Third, we investigate the user features used to personalize on Google Web Search and Bing. Surprisingly, we only nd measurable personalization as a result of searching with a logged in account and the IP address of the searching user. Our results are a rst step towards understanding the extent and e ects of personalization on Web search engines today.

Exposing Inconsistent Web Search Results with Bobble

“…In particular, we treat the search queries (and corresponding results) independently based on the way that a user issues a search query. In contrast to Hannak et al's prior study [5], we find that the profile-based personalization results in significant inconsistencies. Here, we replicate Hannak et al's experimental method.…”

“…The bulk of their effort focuses on understanding the features leading to personalization, but they also conduct a limited study of real-world personalization by hiring 200 US-based workers to search a fixed set of 120 search terms using their own Google accounts [5]. They find that any given slot in the first page of search results has less than a 12% chance of being personalized.…”

“…Bobble captures a user's search query and reissues it from a subset of over 300 world-wide vantage points, including both dedicated PlanetLab measurement nodes and the hosts of other consenting Bobble users. In contrast to research tools that have been developed to measure search personalization offline [5], we intend users to use Bobble while they browse the Web, providing them critical insight into how their online experience is being potentially distorted by personalization.…”

“…We quantify on how geography and search history may influence search results, but others have noted that many other factors (e.g., device type, time of day) may also affect the results that a user sees for a given search term [5]. Bobble has been deployed and publicly available for 21 months; users and researchers can extend it to measure how other factors might induce inconsistencies in search results.…”

TRAP: Using Targeted ads to unveil Google personal profiles

“…While the patents describe the big ads picture, they however do not reveal details about the level of personalization in Google search and displayed ads. Relevant personalization in query results and ads would lead to concerns about the "Filter Bubble" effect, where users are trapped into information bubbles because the search engine algorithms decide such information is relevant for them [12], [13]. In [14], the authors present a measurement methodology for determining the effectiveness of personalized online advertising.…”