Reproducibility and Replicability of Web Measurement Studies

Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert

doi:10.1145/3485447.3512214

Cited by 19 publications

(9 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to other web measurement research, our study has several limitations regarding representativeness, reproducibility [63] and coverage. With a single form factor (desktop) and vantage point in the US, we cannot claim our results are representative of real users' experiences around the world.…”

Section: Limitationsmentioning

confidence: 93%

Unveiling the Impact of User-Agent Reduction and Client Hints: A Measurement Study

Senol,

Acar

2023

Proceedings of the 22nd Workshop on Privacy in the Electronic Society

View full text Add to dashboard Cite

The user-agent string contains the details of a user's device, browser and platform. Prior work on browser fingerprinting showed that the user-agent string can facilitate covert fingerprinting and tracking of users. In order to address these privacy concerns, browsers including Chrome recently reduced the user-agent string to make it less identifying. Simultaneously, Chrome introduced several highly identifying (or high-entropy) user-agent client hints (UA-CH) to allow access to browser properties that are redacted from the useragent string. In this empirical study, we attempt to characterize the effects of these major changes through a large-scale web measurement on the top 100K websites. Using an instrumented crawler, we quantify access to high-entropy browser features through UA-CH HTTP headers and the JavaScript API. We measure access delegation to third parties and investigate whether the new client hints are already used by tracking, advertising and browser fingerprinting scripts. Our results show that high-entropy UA-CHs are accessed by one or more scripts on 59.2% of the successfully visited sites and 93.8% of these calls were made by tracking and advertising-related scripts-primarily by those owned by Google. Overall, we find that scripts from ∼9K distinct registrable (eTLD+1) third-party domains take advantage of their unfettered access and retrieve the high-entropy UA-CHs. We find that on 91.6% of the sites where high-entropy client hints are accessed via the JavaScript API, the high-entropy hints are exfiltrated by a tracker script to a remote server. Turning to high-entropy UA-CHs sent in the HTTP headers-which require opt-in or delegation-we found very limited use. Only 1.3% of the websites use the Accept-CH header to receive high-entropy UA-CHs; and an even smaller fraction of websites (0.4%) delegate high-entropy hints to third-party domains. Overall, our findings indicate that user-agent reduction efforts were effective in minimizing the passive collection of identifying browser features, but third-party tracking and advertising scripts continue to enjoy their unfettered access. CCS CONCEPTS• Security and privacy → Web application security; • General and reference → Measurement.

show abstract

Section: Limitationsmentioning

confidence: 93%

Unveiling the Impact of User-Agent Reduction and Client Hints: A Measurement Study

Senol,

Acar

2023

Proceedings of the 22nd Workshop on Privacy in the Electronic Society

View full text Add to dashboard Cite

show abstract

“…Recently, Nguyen et al [78] studied the implementation of consent notices specifically on Android apps and identified that about 20% of these apps violate at least one GDPR consent. In the study conducted by Demir et al [62], a thorough examination of 114 prior research papers on web measurement was undertaken. The findings revealed a significant trend, with the majority (72.6%) of these papers lacking a comprehensive inventory of the pages they had analyzed.…”

Section: Related Workmentioning

confidence: 99%

Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?

Liu,

Iqbal,

Saxena

2024

PoPETs

View full text Add to dashboard Cite

Data protection regulations, such as GDPR and CCPA, require websites and embedded third-parties, especially advertisers, to seek user consent before they can collect and process user data. Only when the users opt in, should these entities collect, process, and share user data. Websites typically incorporate Consent Management Platforms (CMPs), such as OneTrust and CookieBot, to solicit and convey user consent to the embedded advertisers, with the expectation that the consent will be respected. However, neither the websites nor the regulators currently have any mechanism to audit advertisers' compliance with the user consent, i.e., to determine if advertisers indeed do not collect, process, and share user data when the user opts out. In this paper, we propose an auditing framework that leverages advertisers' bidding behavior to empirically assess the violations of data protection regulations. Using our framework, we conduct a measurement study to evaluate four of the most widely deployed CMPs, i.e., Didomi, Quantcast, OneTrust, and CookieBot, as well as advertiser-offered opt-out controls, i.e., National Advertising Initiative's opt-out, under GDPR and CCPA. Our results indicate that in many cases user data is unfortunately still being collected, processed, and shared even when users opt-out. We also find that some CMPs are better than the others at conveying user consent and that several ad platforms ignore user consent. Our results also indicate that advertiser-offered opt-out are equally ineffective at protecting user privacy.

show abstract

“…The measurement framework provided by Demir et al [9] is the basis for our setup, which allows orchestrating quasi-parallel web measurements using different browser setups. The framework consists of a master machine that starts multiple virtual machines (VM), each running a separate crawler with a distinct configuration.…”

Section: Measurement Setupmentioning

confidence: 99%

“…Previous work has shown that subpages behave differently than the respective landing page [2,9,42]. Therefore, to build the final set of pages to visit, we visit each randomly selected site's landing page and collect 15 subpages (i.e., first-party links on the page) for each.…”

Section: Datasetmentioning

confidence: 99%

A Large-Scale Study of Cookie Banner Interaction Tools and their Impact on Users' Privacy

Demir,

Urban,

Pohlmann

et al. 2024

PoPETs

View full text Add to dashboard Cite

Cookie notices (or cookie banners) are a popular mechanism for websites to provide (European) Internet users a tool to choose which cookies the site may set. Banner implementations range from merely providing information that a site uses cookies over offering the choice to accepting or denying all cookies to allowing fine-grained control of cookie usage. Users frequently get annoyed by the banner's pervasiveness as they interrupt ''natural'' browsing on the Web. As a remedy, different browser extensions have been developed to automate the interaction with cookie banners. In this work, we perform a large-scale measurement study comparing the effectiveness of extensions for ''cookie banner interaction.'' We configured the extensions to express different privacy choices (e.g., accepting all cookies, accepting functional cookies, or rejecting all cookies) to understand their capabilities to execute a user's preferences. The results show statistically significant differences in which cookies are set, how many of them are set, and which types are set---even for extensions that aim to implement the same cookie choice. Extensions for ''cookie banner interaction'' can effectively reduce the number of set cookies compared to no interaction with the banners. However, all extensions increase the tracking requests significantly except when rejecting all cookies.

show abstract

Reproducibility and Replicability of Web Measurement Studies

Cited by 19 publications

References 42 publications

Unveiling the Impact of User-Agent Reduction and Client Hints: A Measurement Study

Unveiling the Impact of User-Agent Reduction and Client Hints: A Measurement Study

Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?

A Large-Scale Study of Cookie Banner Interaction Tools and their Impact on Users' Privacy

Contact Info

Product

Resources

About