Fingerprint Surface-Based Detection of Web Bot Detectors

Abstract

Web bots are used to automate client interactions with websites, which facilitates large-scale web measurements. However, websites may employ web bot detection. When they do, their response to a bot may differ from responses to regular browsers. The discrimination can result in deviating content, restriction of resources or even the exclusion of a bot from a website. This places strict restrictions upon studies: the more bot detection takes place, the more results must be manually verified to confirm the bot's findings. To investigate the extent to which bot detection occurs, we reverseanalysed commercial bot detection. We found that in part, bot detection relies on the values of browser properties and the presence of certain objects in the browser's DOM model. This part strongly resembles browser fingerprinting. We leveraged this for a generic approach to detect web bot detection: we identify what part of the browser fingerprint of a web bot uniquely identifies it as a web bot by contrasting its fingerprint with those of regular browsers. This leads to the fingerprint surface of a web bot. Any website accessing the fingerprint surface is then accessing a part unique to bots, and thus engaging in bot detection. We provide a characterisation of the fingerprint surface of 14 web bots. We show that the vast majority of these frameworks are uniquely identifiable through well-known fingerprinting techniques. We design a scanner to detect web bot detection based on the reverse analysis, augmented with the found fingerprint surfaces. In a scan of the Alexa Top 1 Million, we find that 12.8% of websites show indications of web bot detection.

Beyond the Front Page:Measuring Third Party Dynamics in the Field

“…Aside from our bot mitigation techniques, we do not interact with the visited websites in any way, limitations of this approach are discussed in Section 6. While a website might detect our crawler, it is not detected by current mechanisms seen in the wild, as presented by Jonker et al [24].…”

Shepherd: a Generic Approach to Automating Website Login

“…[EN16]); secondly, such deviations will (also) affect logging in (cf. [JKV19]). Thus, for this project we require an automated way to use a regular (full) browser.…”

Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics

“…), which makes their detection more challenging. Even though such web bots, in their vanilla configurations, contain fingerprints that can reveal their bot nature [23], additional configurations can be applied to avoid detection [22,23]. Moreover, web bots can be designed to use the regular browsers of a machine (instead of using an automated browsing software), which makes fingerprint-based detection even harder [2].…”

“…Such web bots can crawl web servers in a humanlike manner to collect information making them harder to detect. Additionally, we can assume that the malicious web bots exhibit a fingerprint that is indistinguishable from that of a browser as in the opposite case, such bots could be deterministically detected using advanced fingerprinting techniques [5,22,23,31]. This is a logical assumption, as the respective Indicators of Compromise have a low pain threshold (i.e., they require low effort to be changed) [8].…”