What did you see? A study to measure personalization in Google’s search engine

Abstract

In this paper we present the results of the project "#Datenspende" where during the German election in 2017 more than 4000 people contributed their search results regarding keywords connected to the German election campaign. Analyzing the donated result lists we prove, that the room for personalization of the search results is very small. Thus the opportunity for the effect mentioned in Eli Pariser's filter bubble theory to occur in this data is also very small, to a degree that it is negligible. We achieved these results by applying various similarity measures to the result lists that were donated. The first approach using the number of common results as a similarity measure showed that the space for personalization is less than two results out of ten on average when searching for persons and at most four regarding the search for parties. Application of other, more specific measures show that the space is indeed smaller, so that the presence of filter bubbles is not evident. Moreover this project is also a proof of concept, as it enables society to permanently monitor a search engine's degree of personalization for any desired search terms. The general design can also be transferred to intermediaries, if appropriate APIs restrict selective access to contents relevant to the study in order to establish a similar degree of trustworthiness.

Whose relevance? Web search engines as multisided relevance machines

“…As personalization is seen to increase individual relevance at the price of decreasing societal relevance, it could be wise to avoid an excess focus on personalization. Indeed, while the negative effects of personalization have been frequently lamented since the early 2010s (e.g., Pariser, 2011), empirical research has identified the effects of contextualization (e.g., preferring local results), but not many effects from personalization, in Google's top web and news results (Krafft et al, 2019; Nechushtai & Lewis, 2019), although evidence is still too weak to make conclusions. Change the presentation of the results by adding information from trusted sources to the SERP, making organic results disappear from the immediately visible area of the results page. This allows the search engine to have more control over what users will see, as the results in the knowledge panels are not generated algorithmically from an index of web pages but from one or a few manually selected sources only.…”

“…As personalization is seen to increase individual relevance at the price of decreasing societal relevance, it could be wise to avoid an excess focus on personalization. Indeed, while the negative effects of personalization have been frequently lamented since the early 2010s (e.g., Pariser, 2011), empirical research has identified the effects of contextualization (e.g., preferring local results), but not many effects from personalization, in Google's top web and news results (Krafft et al, 2019;Nechushtai & Lewis, 2019), although evidence is still too weak to make conclusions. 4.…”

The algorithmic persuasion framework in online communication: conceptualization and a future research agenda

“…A second method is the use of a custom-built browser plug-in designed to collect participants' online interactions and behaviors (e.g., Bodo et al, 2017;Haim and Nienierza, 2019;Strycharz et al, 2019). This approach has also been referred to as "data donation" (Krafft et al, 2019). A browser plug-in allows researchers to unobtrusively track which pieces of algorithm-mediated information participants are exposed to on their own devices.…”

The Concentration-after-Personalisation Index (CAPI): Governing effects of personalisation using the example of targeted online advertising

“…In recent years, concerns about the possible negative effects of personalised online services have led to efforts to quantify their degree of personalisation. For example, personalisation in web search as well as online news has been measured by using similarity ratios (Cozza et al, 2016; Dos Santos et al, 2020; Hannák et al, 2017; Krafft et al, 2019; Le et al, 2019; Puschmann, 2018; Salehi et al, 2015). Similarity ratios have also been applied to measuring targeted online advertising (Balebako et al, 2012; Guha et al, 2010).…”