Automated Discovery of Internet Censorship by Web Crawling

Abstract

Censorship of the Internet is widespread around the world. As access to the web becomes increasingly ubiquitous, filtering of this resource becomes more pervasive. Transparency about specific content that citizens are denied access to is atypical. To counter this, numerous techniques for maintaining URL filter lists have been proposed by various individuals and organisations that aim to empirical data on censorship for benefit of the public and wider censorship research community.We present a new approach for discovering filtered domains in different countries. This method is fully automated and requires no human interaction. The system uses web crawling techniques to traverse between filtered sites and implements a robust method for determining if a domain is filtered. We demonstrate the effectiveness of the approach by running experiments to search for filtered content in four different censorship regimes. Our results show that we perform better than the current state of the art and have built domain filter lists an order of magnitude larger than the most widely available public lists as of Jan 2018. Further, we build a dataset mapping the interlinking nature of blocked content between domains and exhibit the tightly networked nature of censored web resources. CCS CONCEPTS• Social and professional topics → Censorship; • Security and privacy → Social aspects of security and privacy; • Networks → Naming and addressing;

Analysing Censorship Circumvention with VPNs Via DNS Cache Snooping

“…Censored domains -The second stage of the experiment we repeated against popular domains that we knew to be censored. We gathered these domains using the technique described by Darer et al [7] [8]. This list was also based on backlinks from web pages, similar to the Majestic Million, and so domains with more links to it from other sites were chosen over those domains that only had a few backlinks.…”

On Identifying Anomalies in Tor Usage with Applications in Detecting Internet Censorship

“…This approach aims to produce a continually-updated list of blocked terms that could be used to maintain an understanding of those terms most offensive to the filtering authorities. Similarly, Darer et al [9,10] have used keyword-and crawling-based approaches to discover previously unindentified blocked domains.…”