Cloud Based Web Scraping for Big Data Applications

Chaulagain, Ram Sharan; Pandey, Santosh; Basnet, Sadhu Ram; Shakya, Subarna

doi:10.1109/smartcloud.2017.28

Cited by 46 publications

(34 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Chaulagain et al 8 talks about “selenium” and compares it with other cloud‐based web scraping tools. The architecture of the Web Scraper is compatible with big data applications.…”

Section: Related Workmentioning

confidence: 99%

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Khan

Tsaramirsis

Ullah³

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary Web scraping is the process of extracting data from web pages and it is an essential part for the generation of datasets. Currently the field is dominated by capable commercial applications, however, there is always a need for web crawling and web scraping applications for custom projects. Developing fit for purpose tools for retrieving and structuring data from web services, cloud systems, and big data is a challenging task. Based on empirical studies, some of the challenges include structural issues, formatting/ presentation, availability, denial of service, size, and information fetching problems with browsers. Additionally, the data become inaccessible after the structure/template of the website changes for example, after the website update. Thus the dataset cannot be updated in the future without manually modifying the parameters of the Web Scraper. In this paper we propose an algorithm capable of autocorrecting the template (web scraping parameters) used for locating the target data and dealing with some common empirical problems. This is very useful in case there is a need for updating the dataset later, as usually, websites tend to change their pages. Moreover, we introduce an implementation of the algorithm via a tool developed for extracting data from the unity asset store. The tool can capture and store data in XML format. The tool extracted a total of 46 785 (40 611 3D and 6174 2D) items, with 35 successful first retries, 11 second retries and 5 fails.

show abstract

“…Chaulagain et al 8 talks about “selenium” and compares it with other cloud‐based web scraping tools. The architecture of the Web Scraper is compatible with big data applications.…”

Section: Related Workmentioning

confidence: 99%

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Khan

Tsaramirsis

Ullah³

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…In fact, traditional data collection methods (e.g., questionnaire-based surveys) can be implemented through the Internet [1]. In addition, a huge amount of data can be retrieved from social networks [2][3][4] or by means of web-scraping techniques [5]. However, can these data be used as a valid, truthful, and reliable support in making decisions or in supporting research in various fields?…”

Section: Introductionmentioning

confidence: 99%

Comparing Methods to Collect and Geolocate Tweets in Great Britain

Schlosser

Toninelli

Cameletti

2021

Journal of Open Innovation: Technology, Market, and Complexity

View full text Add to dashboard Cite

In the era of Big Data, the Internet has become one of the main data sources: Data can be collected for relatively low costs and can be used for a wide range of purposes. To be able to timely support solid decisions in any field, it is essential to increase data production efficiency, data accuracy, and reliability. In this framework, our paper aims at identifying an optimized and flexible method to collect and, at the same time, geolocate social media information over a whole country. In particular, the target of this paper is to compare three alternative methods to collect data from the social media Twitter. This is achieved considering four main comparison criteria: Collection time, dataset size, pre-processing phase load, and geographic distribution. Our findings regarding Great Britain identify one of these methods as the best option, since it is able to collect both the highest number of tweets per hour and the highest percentage of unique tweets per hour. Furthermore, this method reduces the computational effort needed to pre-process the collected tweets (e.g., showing the lowest collection times and the lowest number of duplicates within the geographical areas) and enhances the territorial coverage (if compared to the population distribution). At the same time, the effort required to set up this method is feasible and less prone to the arbitrary decisions of the researcher.

show abstract

“…There are various data sources on the internet that can be used in the research process. The process of taking data or information from sites on the internet is called web scraping [2], [3], [4], [5], [6], [7], web extraction [8], [9], [10], [11], web harvesting [12], [13]. Web scraping has been used widely and for different purposes including online price comparison, weather data monitoring, website change detection, research, integrating data from multiple sources, extract offers and discounts, scrape job postings information from job portals, brand monitoring, collect government data and market analysis [14].…”

Section: Introductionmentioning

confidence: 99%

Comparison of Web Scraping Techniques : Regular Expression, HTML DOM and Xpath

Gunawan¹,

Rahmatulloh²,

Darmawan³

et al. 2019

Proceedings of the 2018 International Conference on Industrial Enterprise and System Engineering (IcoIESE 2018)

View full text Add to dashboard Cite

Data collection is the initial stage of research. There are various data sources on the internet that can be used in the research process. The process of taking data or information from sites on the internet is called web scraping. Some methods of web scraping include Regular Expression (Regex), HTML DOM and XPath. This study aims to determine the performance of the three methods of web scraping. The Comparison is done by testing each method when retrieving data from the target website, then measuring the performance of the process and comparing it. Process time, memory usage, and data consumption are used as measurement parameters in the experiment. The results of the experiment show that web scraping with the regex method is the smallest in memory usage compared to the HTML DOM method, and Xpath. While HTML DOM requires the least amount of time and the smallest data consumption compared to Regular Expression and XPath methods.

show abstract

Cloud Based Web Scraping for Big Data Applications

Cited by 46 publications

References 1 publication

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Smart algorithmic based web crawling and scraping with template autoupdate capabilities

Comparing Methods to Collect and Geolocate Tweets in Great Britain

Comparison of Web Scraping Techniques : Regular Expression, HTML DOM and Xpath

Contact Info

Product

Resources

About