Big Data, Big Noise: The Challenge of Finding Issue Networks on the Web

Abstract

In this paper, we focus on noise in the sense of irrelevant information in a data set as a specific methodological challenge of web research in the era of big data. We empirically evaluate several methods for filtering hyperlink networks in order to reconstruct networks that contain only web pages that deal with a particular issue. The test corpus of web pages was collected from hyperlink networks on the issue of food safety in the United States and Germany. We applied three filtering strategies and evaluated their performance to exclude irrelevant content from the networks: keyword filtering, automated document classification with a machine-learning algorithm, and extraction of core networks with network-analytical measures. Keyword filtering and automated classification of web pages were the most effective methods for reducing noise whereas extracting a core network did not yield satisfying results for this case.

Quality of data sets that feed AI and big data applications for law enforcement

“…Without a proper implementation of the data minimisation principle and the specific safeguards contained in the Europol Regulation, data subjects run the risk of wrongfully being linked to a criminal activity across the EU, with all of the potential damage for their personal and family life, freedom of movement and occupation that this entails. 30 The big data challenge was the subject of further consultations between Europol and the EDPS, which resulted in, inter alia, introducing a pre-analysis of personal data received with the sole purpose of determining whether such data fall into the categories of data subjects. This, however, has raised further concerns relating to maximum retention period for data sets lacking Data Subject Categorisation, 31 and the consultations in this regard are still ongoing.…”

“…29 European Data Protection Supervisor [10], Point 4.7. 30 European Data Protection Supervisor [10], Point 4.10. 31 European Data Protection Supervisor [11].…”

“…See the survey of existing definitions: De Mauro, Greco, Grimaldi[5], p. 6-7 15. Waldherr, Maier, Miltner, Günther[30] 16. Wang, Strong[31] 17.…”

Simultaneous clustering and variable selection: A novel algorithm and model selection procedure

“…Although research that exploits data sets with many variables to identify subgroups is promising, it also comes with challenges. One of the most compelling challenges, as stressed by a number of scholars (e.g., Yarkoni and Westfall, 2017;Waldherr et al, 2017;Bzdok & Meyer-Lindenberg, 2018), is that these data sets may comprise a large amount of "irrelevant variables" (Fowlkes & Mallows, 1983). They are variables that do not separate clusters well and therefore do not define cluster structure.…”