Web Scrapping: Data Extraction from Websites

Almaqbali, Iqtibas Salim Hilal; Khufairi, Fatmah Mohammed Ali Al; Khan, Mohamed Samiulla; Bhat, Anjum Zameer; Imran, Ahmed

doi:10.47611/jsr.vi.942

Cited by 11 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Firstly, the pdf files that contain registered pharmaceutical and cosmetic product data were downloaded from the new products approved section on the NPRA website. Then, website links were generated from the product ID retrieved and run in a web scrapping tool, ParseHub [24], to extract all the product information in the search product section. The data obtained from these two sections were merged and stored in excel tables.…”

Section: Pre-process Of Pharmaceutical and Cosmetic Product Informationmentioning

confidence: 99%

MyPharmaceutical: an interactive proof of concept

Jie,

Zaaba,

Omar

2024

Bulletin EEI

View full text Add to dashboard Cite

With the rise of health awareness, pharmaceutical and cosmetic products should be verified to protect ourselves from health risks. MyPharmaceutical is a proof-of-concept proposed to provide a mobile application for users to carry out product verification and reporting and a web application for administrative purposes. The data on the registered pharmaceutical and cosmetic products were extracted from national pharmaceutical regulatory agency (NPRA) website. MyPharmaceutical mobile application provides functionalities such as searching the registered product, bookmarking products, reporting products, and tracking report status. The mobile application also implemented a barcode scanner feature to provide ease of product verification. A named entity recognition algorithm is applied with the NLP.js library to provide an improved product search feature for the users, where products can be searched with multiple search criteria in a single input. The web application is proposed to support the mobile application, where the NPRA data admins and officers can manage reported products, publish announcements, verify product data, and utilize the analytic dashboard. The system proposed is expected to provide ease of product verification and reporting to assist the public in choosing safe registered products and a platform for NPRA to manage data and deliver information to the users.

show abstract

Section: Pre-process Of Pharmaceutical and Cosmetic Product Informationmentioning

confidence: 99%

MyPharmaceutical: an interactive proof of concept

Jie,

Zaaba,

Omar

2024

Bulletin EEI

View full text Add to dashboard Cite

show abstract

“…Web data may be in the form of text, databases, emails, audio, video, images, blogs, tweets, etc., on web pages [26], which creates several technical issues in terms of volume, variety, velocity, and veracity [27]. Business organizations mostly apply decision making applications to a dataset that is collected from the internet for accuracy and faster decision making [28], though internal data (organization's record) are used for analysis with public data (collected from different authentic sources) [29]. Web-scraping is also called data harvesting, screen-scraping, or simply data collecting from the internet [27].…”

Section: Web Data Scrapingmentioning

confidence: 99%

A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data

et al. 2022

View full text Add to dashboard Cite

Business organizations experience cut-throat competition in the e-commerce era, where a smart organization needs to come up with faster innovative ideas to enjoy competitive advantages. A smart user decides from the review information of an online product. Data-driven smart machine learning applications use real data to support immediate decision making. Web scraping technologies support supplying sufficient relevant and up-to-date well-structured data from unstructured data sources like websites. Machine learning applications generate models for in-depth data analysis and decision making. The Internet Movie Database (IMDB) is one of the largest movie databases on the internet. IMDB movie information is applied for statistical analysis, sentiment classification, genre-based clustering, and rating-based clustering with respect to movie release year, budget, etc., for repository dataset. This paper presents a novel clustering model with respect to two different rating systems of IMDB movie data. This work contributes to the three areas: (i) the “grey area” of web scraping to extract data for research purposes; (ii) statistical analysis to correlate required data fields and understanding purposes of implementation machine learning, (iii) k-means clustering is applied for movie critics rank (Metascore) and users’ star rank (Rating). Different python libraries are used for web data scraping, data analysis, data visualization, and k-means clustering application. Only 42.4% of records were accepted from the extracted dataset for research purposes after cleaning. Statistical analysis showed that votes, ratings, Metascore have a linear relationship, while random characteristics are observed for income of the movie. On the other hand, experts’ feedback (Metascore) and customers’ feedback (Rating) are negatively correlated (−0.0384) due to the biasness of additional features like genre, actors, budget, etc. Both rankings have a nonlinear relationship with the income of the movies. Six optimal clusters were selected by elbow technique and the calculated silhouette score is 0.4926 for the proposed k-means clustering model and we found that only one cluster is in the logical relationship of two rankings systems.

show abstract

“…Web scraping is an automated technique or software of extracting interesting data from websites. This method generally centres around to the change of unstructured or massive data (HTML/XML documents) on the web into organized information according to user query [1], [2].…”

Section: Introductionmentioning

confidence: 99%

Optimizing E-Commerce Decision Making Using Web Scraping

Chandrawat,

Kumar,

Bohra

2024

Preprint

View full text Add to dashboard Cite

This research investigates the potential of web scraping techniques to optimize decision-making within the e-commerce landscape. By leveraging advanced scraping methods, the project aims to develop a framework that extracts and analyses data from various e-commerce domains. This data will encompass crucial elements like product information and pricing strategies. Through innovative data analysis techniques, the framework will uncover valuable insights and trends, empowering e-commerce businesses to make informed decisions in several key areas. These areas may include optimizing product selection, implementing dynamic pricing strategies, and crafting targeted marketing campaigns. Ultimately, this research seeks to demonstrate how web scraping, when employed ethically and responsibly, can become a powerful tool for e-commerce businesses to gain a competitive edge and enhance their overall decision-making processes.

show abstract

Web Scrapping: Data Extraction from Websites

Cited by 11 publications

References 0 publications

MyPharmaceutical: an interactive proof of concept

MyPharmaceutical: an interactive proof of concept

A Ranking Learning Model by K-Means Clustering Technique for Web Scraped Movie Data

Optimizing E-Commerce Decision Making Using Web Scraping

Contact Info

Product

Resources

About