Measuring the Degree of Divergence when Labeling Tweets in the Electoral Scenario

Santos, Jéssica Soares dos; Bernardini, Flávia; Paes, Aline

doi:10.5753/brasnam.2021.16131

Cited by 7 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reliability analysis of the gold-standard sets conducted in Section 4.3 confirmed that labeling short online texts is a challenging and often confusing task for human annotators (see [31] but also [58]). At the same time, the inter-annotator agreement achieved in this study for the 9 classes is considerably better than in many other reports which dealt with multiple-class annotation of short texts (e.g., [59,60]).…”

Section: Labelingmentioning

confidence: 78%

“…As a rule, the developed LDA-based approaches require label-topics or seed keywords to be manually selected in the beginning of the annotation process. The latter may be non-trivial, if feasible at all, in the case of e-commerce data, as multiple-topic manual labeling of large volumes of short texts was shown to produce highly subjective and contradictory results (e.g., see [31]). Wang et al [32] proposed an optimization algorithm called Adaptive Labeled LDA (AL-LDA) to deal with label disparity.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Text Segmentation Approach for Automated Annotation of Online Customer Reviews, Based on Topic Modeling

2022

View full text Add to dashboard Cite

Online customer review classification and analysis have been recognized as an important problem in many domains, such as business intelligence, marketing, and e-governance. To solve this problem, a variety of machine learning methods was developed in the past decade. Existing methods, however, either rely on human labeling or have high computing cost, or both. This makes them a poor fit to deal with dynamic and ever-growing collections of short but semantically noisy texts of customer reviews. In the present study, the problem of multi-topic online review clustering is addressed by generating high quality bronze-standard labeled sets for training efficient classifier models. A novel unsupervised algorithm is developed to break reviews into sequential semantically homogeneous segments. Segment data is then used to fine-tune a Latent Dirichlet Allocation (LDA) model obtained for the reviews, and to classify them along categories detected through topic modeling. After testing the segmentation algorithm on a benchmark text collection, it was successfully applied in a case study of tourism review classification. In all experiments conducted, the proposed approach produced results similar to or better than baseline methods. The paper critically discusses the main findings and paves ways for future work.

show abstract

Section: Labelingmentioning

confidence: 78%

Section: Related Workmentioning

confidence: 99%

A Text Segmentation Approach for Automated Annotation of Online Customer Reviews, Based on Topic Modeling

2022

View full text Add to dashboard Cite

show abstract

“…The judgement of whether the tweet fitted into one those categories was based on subjectively analysing the individual tweet without the aid of any other details to remain objective. It is essential that the reliability of these scores be measured (Bobicev & Sokolova, 2017;Landis & Koch, 1977;Santos, Bernardini, Paes, 2021). This can be achieved by other researchers to re-classify the same sample of data to determine the degree of agreement among raters.…”

Section: Manual Classificationmentioning

confidence: 99%

“…This can be achieved by other researchers to re-classify the same sample of data to determine the degree of agreement among raters. This is essential for measuring the reliability of the classified data and also the reliability of the algorithms (Bobicev & Sokolova, 2017;Landis & Koch, 1977;Santos, Bernardini, Paes, 2021). The manual classification of each dataset will be rated by three manual raters, which are known as MR1 (Manual Rater (MR)), MR2, and MR3.…”

Section: Manual Classificationmentioning

confidence: 99%

“…If the raters are in disagreement, then the scale may be defective or the rater requires re-training, so the task is effectively carried out. There are a range of statistics that can be applied for the inter-rater agreement, such as Cohen's kappa and Krippendorff's alpha (Bobicev & Sokolova, 2017;Landis & Koch, 1977;Santos, Bernardini, Paes, 2021). The comparison of the raters will make use of these, which are listed below together with an indication of how they should be interpreted:…”

Section: Manual Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Analysing social media data using sentiment analysis in relation to public order

Baldwin¹

View full text Add to dashboard Cite

The research aim is to analyse social media data using sentiment analysis in relation to public order. A sentiment can be expressed in a thought, opinion or attitude that is mainly based on emotion instead of reason. (SA) Sentiment Analysis studies the opinions, sentiments and emotions expressed at sentence or document level. SA extracts text which is identified and classified as opinions or emotions that aim to support a decision-making process through the analysis of text. SA identifies and measures whether the text being analysed is positive, negative or neutral in relation to an entity, such as people, organisation, event, location, or a topic. As the adoption of ubiquitous technology increases and the population on social media continues to grow with the speed of responsiveness of the users expressing their political, economic or religious views on Twitter or Facebook, the posts become valuable sources of public opinion. This can be seen as an important commodity to be used to infer public opinions for social studies or marketing. The research suggests the police have found it difficult to adapt their existing model to the changing nature of public events and handling of acceleration towards technology and social media. The scalability and volume of data has made it increasingly hard for the police to manage, monitor and make use of intelligence emerging from social media to maintain the peace. To address this gap, the investigation will evaluate whether SA can enhance the analysis of social media in the context of public (dis)order events. This may help to improve the police’s decision-making process and reduce complexity to increase public safety. There are specific and generalised ways that SA can support the police, but this research might focus on a specific case. To meet the aim, the research proposes to use a SA model, data mining tools and techniques to analyse the relevant data extracted from social media. The project will use an adapted social media lifecycle as a methodological approach. Past events involving public order and the police will be evaluated to develop relevant methodology and provide appropriate recommendations to the technical community on ways to use SA for future applications of social media. In the project it adopted a hybrid approach which consists of a dictionary, machine learning and gold standard approaches. As result, the machine learning of dictionaries and manual classification results proved to show the strongest output based on precision, recall and F1 measure when compared to the machine learning of tweets and manual classification. The change point analysis helped to identify significant points in the timeline of tweets for the event which correlated to the physical event. However, there were some inaccuracies on the allocated points of change, as deemed insignificant based on news media and low volume of tweets. Future work is required to understand the reasons behind the allocation change points and possible use of alternative methods to help extract further insights that could not be explored in this project. The study makes a series of contributions to knowledge. First, to the creation of a keywords for public order events due to none being publicly available. Second, is to build towards a model to predict what may happen in public order events with the application of dictionary, machine learning and creation of gold standard in the realm of sentiment analysis. Third, the technical contribution to sentiment analysis community to help provide future recommendations to potentially enhance their framework and what areas require further research in the area. Fourth, is the development of social media lifecycle methodology, which has been tested in this project.

show abstract

Similarity-Based Dataset Recommendation Across Languages and Domains to Sentiment Analysis in the Electoral Domain

Santos¹,

Bernardini²,

Paes³

2022

Electronic Participation

View full text Add to dashboard Cite

Measuring the Degree of Divergence when Labeling Tweets in the Electoral Scenario

Cited by 7 publications

References 13 publications

A Text Segmentation Approach for Automated Annotation of Online Customer Reviews, Based on Topic Modeling

A Text Segmentation Approach for Automated Annotation of Online Customer Reviews, Based on Topic Modeling

Analysing social media data using sentiment analysis in relation to public order

Similarity-Based Dataset Recommendation Across Languages and Domains to Sentiment Analysis in the Electoral Domain

Contact Info

Product

Resources

About