Dark Web—Onion Hidden Service Discovery and Crawling for Profiling Morphing, Unstructured Crime and Vulnerabilities Prediction

Rawat, Romil; Rajawat, Anand Singh; Mahor, Vinod; Shaw, Rabindra Nath; Ghosh, Ankush

doi:10.1007/978-981-16-0749-3_57

Cited by 44 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Discovering Topics from Dark Websites [102] No ACN, like I2P, Freenet, Lokinet, or I2P. Dark Web-Onion Hidden Service Discovery and Crawling for Profiling Morphing, Unstructured Crime and Vulnerabilities Prediction [83] Theoretical work; no crawler used.…”

Section: Title Reference Reason For Exclusion Big Data Methods and Th...mentioning

confidence: 99%

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Bergman

Popov

2023

IEEE Access

View full text Add to dashboard Cite

Strong encryption algorithms and reliable anonymity routing have made cybercrime investigation more challenging. Hence, one option for law enforcement agencies (LEAs) is to search through unencrypted content on the Internet or anonymous communication networks (ACNs). The capability of automatically harvesting web content from web servers enables LEAs to collect and preserve data prone to serve as potential leads, clues, or evidence in an investigation. Although scientific studies have explored the field of web crawling soon after the inception of the web, few research studies have thoroughly scrutinised web crawling on the "dark web" or via ACNs such as I2P, IPFS, Freenet, and Tor. The current paper presents a systematic literature review (SLR) that examines the prevalence and characteristics of dark web crawlers. From a selection of 58 peer-reviewed articles mentioning crawling and the dark web, 34 remained after excluding irrelevant articles. The literature review showed that most dark web crawlers were programmed in Python, using either Selenium or Scrapy as the web scraping library. The knowledge gathered from the systematic literature review was used to develop a Tor-based web crawling model into an already existing software toolset customised for ACN-based investigations. Finally, the performance of the model was examined through a set of experiments. The results indicate that the developed crawler was successful in scraping web content from both clear and dark web pages, and scraping dark marketplaces on the Tor network. The scientific contribution of this paper entails novel knowledge concerning ACN-based web crawlers. Furthermore, it presents a model for crawling and scraping clear and dark websites for the purpose of digital investigations. The conclusions include practical implications of dark web content retrieval and archival, such as investigation clues and evidence, and the related future research topics.INDEX TERMS cybercrime, digital forensics, systematic literature review, dark web crawling, Tor

show abstract

Section: Title Reference Reason For Exclusion Big Data Methods and Th...mentioning

confidence: 99%

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Bergman

Popov

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…One of the advantages of the deep Web for libraries is its ability to offer secure access to sensitive or private information (Hsieh-Yee, 2013). Libraries can use the deep Web to store and share confidential documents, archives or research data, ensuring data security and privacy for both library staff and users (Rawat et al , 2021).…”

Section: Understanding the Deep Web As A Transformative Tool For Libr...mentioning

confidence: 99%

Revolutionizing libraries: leveraging the deep Web as a lasting solution for displacement and replacement challenges

Bashorun,

Ajani,

Fagbola

2023

LHTN

View full text Add to dashboard Cite

Purpose This paper aims to explore the deep Web as a solution for displacement and replacement challenges in libraries, addressing the challenges, benefits, strategies and case studies. Design/methodology/approach The paper synthesizes existing literature on deep Web integration in libraries, providing a comprehensive analysis of insights from scholarly articles, case studies and expert opinions. Findings The deep Web grants libraries access to unique content, improving information access, fostering collaboration and enabling personalized content. However, security, privacy, ethics and data protection must be considered. Originality/value This paper contributes to the literature by providing a comprehensive examination of deep Web integration in libraries, offering valuable recommendations for navigating the changing landscape and leveraging the deep Web’s potential.

show abstract

“…After the process of downloading an illegal deal data is completed, the data is subjected to processing and categorization so that it can be stored for a longer period. The Figure 1 illustrates how a crawler works [36]. A web crawler, which is also known as a web spider, is an internet bot that has the ability to crawl across some HTML websites to collect information about a site.…”

Section: Crawling the Dark Webmentioning

confidence: 99%

Dark Web Illegal Activities Crawling and Classifying Using Data Mining Techniques

Alaidi

Al_airaji

Alrikabi

et al. 2022

Int. J. Interact. Mob. Technol.

View full text Add to dashboard Cite

Dark web is a canopy concept that denotes any kind of illicit activities carried out by anonymous persons or organizations, thereby making it difficult to trace. The illicit content on the dark web is constantly updated and changed. The collection and classification of such illegal activities are challenging tasks, as they are difficult and time-consuming. This problem has in recent times emerged as an issue that requires quick attention from both the industry and academia. To this end, efforts have been made in this article a crawler that is capable of collecting dark web pages, cleaning them, and saving them in a document database, is proposed. The crawler carries out an automatic classification of the gathered web pages into five classes. The classifiers used in classifying the pages include Linear Support Vector Classifier (SVC), Naïve Bayes (NB), and Document Frequency (TF-IDF). The experimental results revealed that an accuracy rate of 92% and 81% were achieved by SVC and NB, respectively.

show abstract

Dark Web—Onion Hidden Service Discovery and Crawling for Profiling Morphing, Unstructured Crime and Vulnerabilities Prediction

Cited by 44 publications

References 32 publications

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Exploring Dark Web Crawlers: A Systematic Literature Review of Dark Web Crawlers and Their Implementation

Revolutionizing libraries: leveraging the deep Web as a lasting solution for displacement and replacement challenges

Dark Web Illegal Activities Crawling and Classifying Using Data Mining Techniques

Contact Info

Product

Resources

About