Focused Crawling Through Reinforcement Learning

Han, Miyoung; Wuillemin, Pierre-Henri; Senellart, Pierre

doi:10.1007/978-3-319-91662-0_20

Cited by 14 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors of [19]proposed an adaptive topical crawler that uses interactive and automatic adaption approaches for online learning of hyperlink selection policy, form previously crawled pages. Han et al utilized reinforcement learning for topical web crawling [20]. They formulated the problem as a Markov decision process and proposed a new representation of states and actions which considers both content information and the link structure.…”

Section: Topical Crawling Methodsmentioning

confidence: 99%

Cost-Sensitive Topical Data Acquisition From the Web

Naghibi¹,

Anvari²,

Forghani³

et al. 2019

IJDKP

View full text Add to dashboard Cite

The cost of acquiring training data instances for induction of data mining models is one of the main concerns in real-world problems. The web is a comprehensive source for many types of data which can be used for data mining tasks. But the distributed and dynamic nature of web dictates the use of solutions which can handle these characteristics. In this paper, we introduce an automatic method for topical data acquisition from the web. We propose a new type of topical crawlers that use a hybrid link context extraction method for topical crawling to acquire on-topic web pages with minimum bandwidth usage and with the lowest cost. The new link context extraction method which is called Block Text Window (BTW), combines a text window method with a block-based method and overcomes challenges of each of these methods using the advantages of the other one. Experimental results show the predominance of BTW in comparison with state of the art automatic topical web data acquisition methods based on standard metrics.

show abstract

Section: Topical Crawling Methodsmentioning

confidence: 99%

Cost-Sensitive Topical Data Acquisition From the Web

Naghibi¹,

Anvari²,

Forghani³

et al. 2019

IJDKP

View full text Add to dashboard Cite

show abstract

“…In our setting, an RL algorithm allows the crawler to determine a strategy (policy) so that it retrieves a fixed number of documents while maximizing the number of related ones. Recently, there have been approaches of focused crawling [17] and biomedical data mining [18] with RL. An agent (the crawler) fetches URLs in an iterative manner.…”

Section: Reinforcement Learningmentioning

confidence: 99%

A Personalized Machine-Learning-Enabled Method for Efficient Research in Ethnopharmacology. The Case of the Southern Balkans and the Coastal Zone of Asia Minor

et al. 2021

View full text Add to dashboard Cite

Ethnopharmacology experts face several challenges when identifying and retrieving documents and resources related to their scientific focus. The volume of sources that need to be monitored, the variety of formats utilized, and the different quality of language use across sources present some of what we call “big data” challenges in the analysis of this data. This study aims to understand if and how experts can be supported effectively through intelligent tools in the task of ethnopharmacological literature research. To this end, we utilize a real case study of ethnopharmacology research aimed at the southern Balkans and the coastal zone of Asia Minor. Thus, we propose a methodology for more efficient research in ethnopharmacology. Our work follows an “expert–apprentice” paradigm in an automatic URL extraction process, through crawling, where the apprentice is a machine learning (ML) algorithm, utilizing a combination of active learning (AL) and reinforcement learning (RL), and the expert is the human researcher. ML-powered research improved the effectiveness and efficiency of the domain expert by 3.1 and 5.14 times, respectively, fetching a total number of 420 relevant ethnopharmacological documents in only 7 h versus an estimated 36 h of human-expert effort. Therefore, utilizing artificial intelligence (AI) tools to support the researcher can boost the efficiency and effectiveness of the identification and retrieval of appropriate documents.

show abstract

“…In our setting, an RL algorithm allows the crawler to determine a strategy (policy), so that it retrieves a fixed number of documents while maximizing the number of related ones. Recently, there have been approaches of focused crawling [16] and biomedical data mining [17] with RL. An agent (the crawler) fetches URLs in an iterative manner.…”

Section: Reinforcement Learningmentioning

confidence: 99%

A Personalized Machine-Learning-enabled Method for Efficient Research in Ethnopharmacology. The case of Southern Balkans and Coastal zone of Asia Minor

Axiotis¹,

Kontogiannis²,

Kalpoutzakis³

et al. 2021

Preprint

View full text Add to dashboard Cite

Ethnopharmacology experts face several challenges when identifying and retrieving documents and resources related to their scientific focus. The volume of sources that need to be monitored, the variety of formats utilized, the different quality of language use across sources, present some of what we call “big data” challenges in the analysis of this data. This study aims to understand if and how experts can be supported effectively through intelligent tools in the task of ethnopharmacological literature research. To this end, we utilize a real case study of ethnopharmacology research, aimed at the Southern Balkans and Coastal zone of Asia Minor. Thus, we propose a methodology for more efficient research in ethnopharmacology. Our work follows an “Expert-Apprentice” paradigm in an automatic URL extraction process, through crawling, where the apprentice is a Machine Learning (ML) algorithm, utilizing a combination of Active Learning (AL) and Reinforcement Learning (RL), and the Expert is the human researcher. ML-powered research improved 3.1 times the effectiveness and 5.14 times the efficiency of the domain expert, fetching a total number of 420 relevant ethnopharmacological documents in only 7 hours versus an estimated 36-hour human-expert effort. Therefore, utilizing Artificial Intelligence (AI) tools to support the researcher can boost the efficiency and effectiveness of the identification and retrieval of appropriate documents.

show abstract

Focused Crawling Through Reinforcement Learning

Cited by 14 publications

References 15 publications

Cost-Sensitive Topical Data Acquisition From the Web

Cost-Sensitive Topical Data Acquisition From the Web

A Personalized Machine-Learning-Enabled Method for Efficient Research in Ethnopharmacology. The Case of the Southern Balkans and the Coastal Zone of Asia Minor

A Personalized Machine-Learning-enabled Method for Efficient Research in Ethnopharmacology. The case of Southern Balkans and Coastal zone of Asia Minor

Contact Info

Product

Resources

About