RCrawler: An R package for parallel web crawling and scraping

Khalil, Salim; Fakir, Mohamed

doi:10.1016/j.softx.2017.04.004

Cited by 64 publications

(37 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Estes bots, fazem rastreamento e raspagem de dados, utilizando duas técnicas, a saber: web crawling e web scraping. Estas técnicas podem ser usadas de forma simultânea ou como duas tarefas distintas [Khalil and Fakir 2017].…”

Section: Bots Da Webunclassified

Utilização de Bots para Obtenção Automática de Dados Públicos usando as Técnicas de Web Crawling e Web Scraping

Galdino¹,

Gallindo²,

Moreira³

2020

Anais Do Workshop De Computação Aplicada Em Governo Eletrônico (WCGE 2020)

View full text Add to dashboard Cite

A Escola Virtual.Gov (EV.G), recebe recursos de instituições parceiras para disponibilizar a oferta de cursos que são demandados por elas. A fins de promover uma transparência ativa e em conformidade com a Lei de Acesso à Informação, a prestação de contas da aplicação desses recursos precisa estar disponível ao cidadão comum. A partir disto, a EV.G gerencia a aplicação dos recursos através de um sistema próprio. Desse modo, a alimentação do sistema é feita de forma manual. Nessa situação, dada a necessidade da EV.G, este trabalho simplifica o processo de atualização do Portal em Números, automatizando as atividades de alimentação manual hoje realizadas pela EV.G e publicando as informações obtidas na fonte de dados do portal.

show abstract

Section: Bots Da Webunclassified

Utilização de Bots para Obtenção Automática de Dados Públicos usando as Técnicas de Web Crawling e Web Scraping

Galdino¹,

Gallindo²,

Moreira³

2020

Anais Do Workshop De Computação Aplicada Em Governo Eletrônico (WCGE 2020)

View full text Add to dashboard Cite

show abstract

“…Web crawling is the most common technique to gather numerous online reviews in the text or picture format on social media [24]. Although crawlers can scrape the target dataset conveniently and continuously from any websites, they consume resources of visited systems and will cause the load and schedule issue [25]. To improve the data extraction efficiency, several crawling tools could be installed and utilized directly, such as Scrapy (scrapy.org), python-requests (docs.python-requests.org), and bazhuayu (www.bazhuayu.com).…”

Section: Social Media Miningmentioning

confidence: 99%

Multifunctional Product Marketing Using Social Media Based on the Variable-Scale Clustering

Wang

Gao

2019

Teh. vjesn.

View full text Add to dashboard Cite

Customers' demands have become more dynamic and complicated owing to the functional diversity and lifecycle reduction of products which pushes enterprises to identify the real-time needs of distinct customers in a superior way. Meanwhile, social media turned as an emerging channel where customers often spontaneously can express their perceptions and thoughts about products promptly. This paper examines the customer satisfaction identification and improvement problem based on social media mining. First, we proposed the public opinion sensitivity index (POSI) to uncover target customers from extensive short-textual reviews. Subsequently, we presented a customer segmentation approach based on the sentiment analysis and the variable-scale clustering (VSC). The approach is able to get several customer clusters with the same satisfaction level where customers belonging to each cluster have similar interests. Finally, customer-centered marketing strategies and customer difference marketing campaigns are planned under the shadow of customer segmentation results. The experiments illustrate that our proposed method can support marketing decision marketing in practice that enriches the intention of the current customer relationship management.

show abstract

“…There are various data sources on the internet that can be used in the research process. The process of taking data or information from sites on the internet is called web scraping [2], [3], [4], [5], [6], [7], web extraction [8], [9], [10], [11], web harvesting [12], [13]. Web scraping has been used widely and for different purposes including online price comparison, weather data monitoring, website change detection, research, integrating data from multiple sources, extract offers and discounts, scrape job postings information from job portals, brand monitoring, collect government data and market analysis [14].…”

Section: Introductionmentioning

confidence: 99%

Comparison of Web Scraping Techniques : Regular Expression, HTML DOM and Xpath

Gunawan¹,

Rahmatulloh²,

Darmawan³

et al. 2019

Proceedings of the 2018 International Conference on Industrial Enterprise and System Engineering (IcoIESE 2018)

View full text Add to dashboard Cite

Data collection is the initial stage of research. There are various data sources on the internet that can be used in the research process. The process of taking data or information from sites on the internet is called web scraping. Some methods of web scraping include Regular Expression (Regex), HTML DOM and XPath. This study aims to determine the performance of the three methods of web scraping. The Comparison is done by testing each method when retrieving data from the target website, then measuring the performance of the process and comparing it. Process time, memory usage, and data consumption are used as measurement parameters in the experiment. The results of the experiment show that web scraping with the regex method is the smallest in memory usage compared to the HTML DOM method, and Xpath. While HTML DOM requires the least amount of time and the smallest data consumption compared to Regular Expression and XPath methods.

show abstract

RCrawler: An R package for parallel web crawling and scraping

Cited by 64 publications

References 6 publications

Utilização de Bots para Obtenção Automática de Dados Públicos usando as Técnicas de Web Crawling e Web Scraping

Utilização de Bots para Obtenção Automática de Dados Públicos usando as Técnicas de Web Crawling e Web Scraping

Multifunctional Product Marketing Using Social Media Based on the Variable-Scale Clustering

Comparison of Web Scraping Techniques : Regular Expression, HTML DOM and Xpath

Contact Info

Product

Resources

About