Web Scraping: State-of-the-Art and Areas of Application

Diouf, Rabiyatou; Sarr, Edouard Ngor; Sall, Ousmane; Birregah, Babiga; Bousso, Mamadou; Mbaye, Seny Ndiaye

doi:10.1109/bigdata47090.2019.9005594

Cited by 88 publications

(48 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Unit testing frameworks. Although Selenium can be used as a standalone framework (for example, when making web scrapping to gather data from websites [30]), it is a common practice to embed a Selenium script inside a unit test. This way, the execution of the Selenium logic results in a test verdict (pass or fail).…”

mentioning

confidence: 99%

A Survey of the Selenium Ecosystem

et al. 2020

View full text Add to dashboard Cite

Selenium is often considered the de-facto standard framework for end-to-end web testing nowadays. It allows practitioners to drive web browsers (such as Chrome, Firefox, Edge, or Opera) in an automated fashion using different language bindings (such as Java, Python, or JavaScript, among others). The term ecosystem, referring to the open-source software domain, includes various components, tools, and other interrelated elements sharing the same technological background. This article presents a descriptive survey aimed to understand how the community uses Selenium and its ecosystem. This survey is structured in seven categories: Selenium foundations, test development, system under test, test infrastructure, other frameworks, community, and personal experience. In light of the current state of Selenium, we analyze future challenges and opportunities around it.

show abstract

mentioning

confidence: 99%

A Survey of the Selenium Ecosystem

et al. 2020

View full text Add to dashboard Cite

show abstract

“…For implementation and better result purpose we created a dataset a CSV file we made it by using web scraping method we scraped the news articles for authentic news web sites and created a dataset having more than 20,000 news here below is the screenshot of the result of making the dataset using web scraping [9] [10].…”

Section: Implementation and Resultsmentioning

confidence: 99%

Fake News Detection using Machine Learning

Kulkarni¹,

Karwande²,

Keskar³

et al. 2021

ITM Web Conf.

View full text Add to dashboard Cite

Everyone depends upon various online resources for news in this modern age, where the internet is pervasive. As the use of social media platforms such as Facebook, Twitter, and others has increased, news spreads quickly among millions of users in a short time. The consequences of Fake news are far-reaching, from swaying election outcomes in favor of certain candidates to creating biased opinions. WhatsApp, Instagram, and many other social media platforms are the main source for spreading fake news. This work provides a solution by introducing a fake news detection model using machine learning. This model requires prerequisite data extracted from various news websites. Web scraping technique is used for data extraction which is further used to create datasets. The data is classified into two major categories which are true dataset and false dataset. Classifiers used for the classification of data are Random Forest, Logistic Regression, Decision Tree, KNN and Gradient Booster. Based on the output received the data is classified either as true or false data. Based on that, the user can find out whether the given news is fake or not on the webserver.

show abstract

“…This exhaustive dataset will help us analyze the linguistic characteristics and train the algorithm in a much more efficient and accurate manner 3) Data scraping: Web scraping is basically the process of extraction and collection of data from various web pages. Instead of collecting this data manually, we build a web scraping engine that can extract significant amounts of data in a very short span of time [10]. The web scraping engine consists of a crawler and a scraper.The web crawler browses the internet and tries to locate all the relevant content required.…”

Section: A Dataset Preparationmentioning

confidence: 99%

Combating Misinformation Dissemination through Verification and Content Driven Recommendation

Hawa

Lobo

Dogra

et al. 2021

2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV)

View full text Add to dashboard Cite

The COVID 19 pandemic is a humanitarian emergency that poses an enormous threat to society and has impacted various social media platforms and journalism. News and social media has become an immensely popular platform for consumption of information. However, these platforms are also the bearer of fake news and information which causes negative effects and creates panic. Thus, this research work aim to tackle this problem by creating a unique hybrid model using Machine learning algorithms with Natural Language Processing (NLP) techniques to verify news. In order to make the proposed system foolproof, a superior content based recommendation system is developed which will encourage users to consume authenticated news and content from verified sources. Thus, such a system will provide a holistic approach as it not only verifies but also provides genuine and true recommendations for the same.

show abstract

Web Scraping: State-of-the-Art and Areas of Application

Cited by 88 publications

References 20 publications

A Survey of the Selenium Ecosystem

A Survey of the Selenium Ecosystem

Fake News Detection using Machine Learning

Combating Misinformation Dissemination through Verification and Content Driven Recommendation

Contact Info

Product

Resources

About