Web Scraping versus Twitter API

Dongo, Irvin; Cadinale, Yudith; Aguilera, Ana; Martinez, Fabiola; Quintero, Yuni; Barrios, Sergio

doi:10.1145/3428757.3429104

Cited by 20 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, the data collection process is carried out using a scraping technique that scrapes tweets associated with the PPKM policy. Scraping is a data extraction technique applied to collect large amounts of relevant metadata, which is then stored in a database or specific files for future analysis [32]. In the research, scraping uses Python programming language with the help of scrape modules.…”

Section: B Data Acquisitionmentioning

confidence: 99%

Tweets Emotions Analysis of Community Activities Restriction as COVID-19 Policy in Indonesia Using Support Vector Machine

Sutranggono¹,

Imah²

2023

CommIT (Communication and Information Technology) Journal

View full text Add to dashboard Cite

With the rising number of COVID-19 cases in Indonesia, the government has implemented the Imposition of Restrictions on Emergency Community Activities (Pemberlakuan Pembatasan Kegiatan Masyarakat - PPKM) as Indonesia’s COVID-19 policy. Several controversies and protests have colored the implementation of this emergency policy. Some netizens on Twitter voice their opinions about the policy in their tweets. Emotions in tweets can be recognized through text-based emotion detection or emotion analysis. However, textbased emotion detection is a challenging task. One of the main issues in classifying text with a machine learningbased approach deals with the feature dimensions. As a result, appropriate methods for accurately identifying emotion based on the text are required. The research studies an emotions analysis task on Indonesians’ PPKMrelated tweets to understand their emotional state while implementing the PPKM. The machine learning classification algorithms used are Support Vector Machine (SVM) and random forest. The total number of tweets is 4,401. The results show that SVM with linear kernel function combined with the TF-IDF and Chi-Square methods outperforms other classifiers with an accuracy of 0.7528. The accuracy value is higher than those obtained by previous studies. Moreover, the results of the emotion classification on PPKM tweets reveal that most Indonesians are unhappy with the implementation of the PPKM policy.

show abstract

Section: B Data Acquisitionmentioning

confidence: 99%

Tweets Emotions Analysis of Community Activities Restriction as COVID-19 Policy in Indonesia Using Support Vector Machine

Sutranggono¹,

Imah²

2023

CommIT (Communication and Information Technology) Journal

View full text Add to dashboard Cite

show abstract

“…A credibility model to analyze social media, based in three levels of credibility (i.e., text, user, and social impact) was proposed in a previous work [27]. This original credibility model was instantiated in T-CREo, a credibility analysis framework for Twitter [28] and used to perform data source comparison [83], [84]. T-CREo works as a Google Chrome extension to perform an automatic credibility analysis using web scraping techniques and the Twitter API to achieve this analysis in real time.…”

Section: A Credibility Model For Twittermentioning

confidence: 99%

CrediBot: Applying Bot Detection for Credibility Analysis on Twitter

Aguilera,

Quinteros,

Dongo

et al. 2023

IEEE Access

Self Cite

View full text Add to dashboard Cite

Nowadays, people and organizations use social networks for allowing and facilitating the transfer of information among groups that share similar interests. Due to the wide repertoire of users that these social platforms have and the amount of information generated within them, the presence of bots has become a relevant issue, both to facilitate the sharing of true information or to disseminate false information (fake news). In the second case, bots could manipulate political opinions, be perpetrators of identity or information theft, among other possible dangers that can cause when interacting on the platform. Thus, the identification of bots in social networks can become a useful practice to evaluate credibility or to detect fake news. In this work, we extend a previously proposed credibility model for Twitter, by incorporating bot detection. The original model calculates the credibility of tweets based on three measures: text, account/user, and social impact, using different filters to analyse text (SPAM, bad words,and good spelling) and account attributes (e.g., creation date, followers, following) to calculate account/user and social credibility scores. The extended model considers in the user credibility, the bot verification. Additionally, the extended credibility model is implemented in T-CREo, a framework for real time credibility analysis. For bot detection, some machine learning algorithms for supervised learning, such as AdaBoost, Bagging, Decision Tree, Logistic Regression, and Random Forest are trained and evaluated. Results show that the best algorithm is the Random Forest for its capacity of generalization with an accuracy and F1-score values over 97% both in English and Spanish. The evaluation of the bot detection functionality in the credibility analysis shows a performance of precision=1.0, recall=0.8462, F1-score=0.9167, and accuracy=0.92 for both English and Spanish models in our validation tests.

show abstract

“…The existence of this web scraping technique can be used for research purposes, data analysis, and information collection from various media on the internet automatically. Examples of the use of web scraping include the acquisition and categorization of web page information related to hydroponics [6], classification of job vacancies based on data search results on the internet [7], and analysis of natural disaster information [8], Twitter web scraping [9] [10], web scraping on GitHub [11]- [14], web scraping data on google scholar [15], web scraping on Instagram [16] [17].…”

Section: Introductionmentioning

confidence: 99%

Evaluating Web Scraping Performance Using XPath, CSS Selector, Regular Expression, and HTML DOM With Multiprocessing Technical Applications

Darmawan

Maulana²,

Gunawan³

et al. 2022

JOIV : Int. J. Inform. Visualization

View full text Add to dashboard Cite

Data collection has become a necessity today, especially since many sources of data on the internet can be used for various needs. The main activity in data collection is collecting quality information that can be analyzed and used to support decisions or provide evidence. The process of retrieving data from the internet is also known as web scraping. There are various methods of web scraping that are commonly used. The amount of data scattered on the internet will be quite time-consuming if the web scraping is done on a large scale. By applying the parallel concept, the multi-processing approach can help complete a job. This study aimed to determine the performance of the web scraping method with the application of multi-processing. Testing is done by doing the process of scraping data from a predetermined target web. Four web scraping methods: CSS Selector, HTML DOM, Regex, and XPath, were selected to be used in the experiment measured based on the parameters of CPU usage, memory usage, execution time, and bandwidth usage. Based on experimental data, the Regex method has the least CPU and memory usage compared to other methods. While XPath requires the least time compared to other methods. The CSS Selector method is the smallest in terms of bandwidth usage compared to other methods. The application of multi-processing techniques to each web scraping method is proven to save memory usage, reduce execution time and reduce bandwidth usage compared to only using single processing.

show abstract

Web Scraping versus Twitter API

Cited by 20 publications

References 19 publications

Tweets Emotions Analysis of Community Activities Restriction as COVID-19 Policy in Indonesia Using Support Vector Machine

Tweets Emotions Analysis of Community Activities Restriction as COVID-19 Policy in Indonesia Using Support Vector Machine

CrediBot: Applying Bot Detection for Credibility Analysis on Twitter

Evaluating Web Scraping Performance Using XPath, CSS Selector, Regular Expression, and HTML DOM With Multiprocessing Technical Applications

Contact Info

Product

Resources

About