To scrape or not to scrape, this is dilemma. The post-API scenario and implications on digital research

Abstract

IntroductionThis article aims to investigate the potential impact of restricted social data access on digital research practices. The 2018 Cambridge Analytica scandal exposed the exploitation of Facebook user data for speculative purposes and led to the end of the so-called “Data Golden Age,” characterized by free access to social media user data. As a result, many social platforms have limited or entirely banned data access. This policy shift, referred to as the “APIcalypse,” has revolutionized digital research methods.MethodsTo address the impact of this policy shift on digital research, a non-probabilistic sample of Italian researchers was surveyed and the responses were analyzed. The survey was designed to explore how constraints on digital data access have altered research practices, whether we are truly in a post-API era with a radical change in data scraping strategies, and what shared and sustainable solutions can be identified for the post-API scenario.ResultsThe findings highlight how limits on social data access have not yet created a “post-Api” scenario as expected, but it is turning research practices upside down, positively and negatively. On the positive side, because researchers are experimenting with innovative forms of scraping. Negatively, because there could be a “mass migration” to the few platforms that freely grant their APIs, with critical consequences for the quality of research.DiscussionThe closure of many social media APIs has not opened up a post-API world, but has worsened the conditions of making research, which is increasingly oriented to “easy-data” environments such as Twitter. This should prompt digital researchers to make a self-reflexive effort to diversify research platforms and especially to act ethically with user data. It would also be important for the scientific world and large platforms to enter into understandings for open and conscious sharing of data in the name of scientific progress.

Studying and Eliciting Self-Disclosure: Interdisciplinary Review of Research Methodologies and Behavioural Paradigms

“…Moreover, with many social media platforms changing their regulations, making it harder to scrape and crawl data 2 and cancelling API programs that facilitated research (see Davidson et al, 2023), researchers are increasingly relying on smaller, manually collected datasets. This shift often involves more qualitative analysis of content (Trezza, 2023) and a greater reliance on data donations to study user activity on these platforms (Davidson et al, 2023;Ohme et al, 2023).…”

“…EMI can be implemented through traditional means of data collection like using instructions and recorder (e.g., LaFreniere & Newman, 2016), over a telephone call (e.g.,Stone et al, 2019), as well as via instant messaging apps. Nonetheless, these interventions can also be automated and mediated through digital platforms (e.g.,Coyle, Doherty, Matthews, & Sharry, 2007), delivering interventions via mobile apps, web-based tools, and even via artificial agents like robots in people's naturalistic settings (e.g.,Laban, Morrison, Kappas, & Cross, 2022, 2023.…”