Interpretation of Sentiment Analysis with Human-in-the-Loop

Yeruva, Vijaya Kumari; Chandrashekar, Mayanka; Lee, Yugyung; Rydberg-Cox, Jeff; Blanton, Virginia; Oyler, Nathan A.

doi:10.1109/bigdata50022.2020.9378221

Cited by 7 publications

(9 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The significance of human-machine collaboration for sentiment analysis and determining the degree of agreement between several human and machine annotators was highlighted in a more recent study called "human-in-the-loop." References [25] and [26] analyzed the performance and agreement between off-theshelf sentiment analysis tools and reported the sentiment measurement done by an average of 5.63 coders proved satisfactory reliability of Krippendorff's alpha = .80, where the assessment was made on a sample of 148 randomly selected news items analyzing the sentiment of newspaper and website headlines, manually annotated by a team of 22 student coders who were initially trained. The research in [27] focused on more expressive annotations by conducting a two-phase annotation arrangement and showed that perceived emotions can be different from expressed emotions in an event-focused corpus, in turn affecting the performance of the classifier.…”

Section: Literature Reviewmentioning

confidence: 99%

Development and Evaluation of Gold Standard Dataset for Sentiment Analysis of Tweets

Ahmed,

Hina,

Asif

et al. 2024

PakJET

View full text Add to dashboard Cite

Pre-labeled data is typically required for supervised machine learning. A limited number of object classes in the majority of open access and pre-annotated datasets make them unsuitable for certain tasks, even though they are readily available for training machine learning algorithms. For custom models, previously available pre-annotated data is typically insufficient, so gathering and preparing training data is necessary for the majority of real-world applications. The quantity and quality of annotations clearly trade-off with one another. Either more annotated data can be produced or better data quality can be guaranteed by allocating time and resources. Development of the gold standard by annotating textual information is an essential part of the “Text Analytics” domain in the field of “Natural Language Processing-NLP”. In “Text Analytics”, annotation can be done by adopting a manual, semi-automatic or automatic approach. In the case of the manual approach, annotators often work with partial parts of the corpus, and the results are generalized by automated text classification which may affect the final classification results. Annotations reliability and suitability of assigned labels are particularly important in the NLP applications related to opinion mining or sentiment analysis. In this research study, we have evaluated the significance of the annotation process on a novel dataset that contained multiple languages (English, Roman Urdu), a free text dataset that was extracted from Twitter. This unique dataset contained multiple languages which makes this annotation process essential for researching this data. Using this multi-language dataset, we examine the inter-annotator agreement in multiclass and multi-label sentiment annotation. To scrutinize the reliability of this research work, several annotation agreement metrics, statistical analysis, and Machine Learning methods have been considered to evaluate the accuracy of resulting annotations. It was observed that the annotation process is significant and a complex step that is essential for the proper implementation of Natural Language processing tasks for text analytics in machine learning. During this research, different gaps were identified and resolved which can impact the overall reliability of the annotation process which are reported in this paper. We conclude that while inaccurate annotations worsen the results, the impact is minimal, at least when using text data. The advantages of the larger annotated data set (obtained by employing subpar auto-annotation techniques) surpass the degradation resulting from the use of annotated data.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Development and Evaluation of Gold Standard Dataset for Sentiment Analysis of Tweets

Ahmed,

Hina,

Asif

et al. 2024

PakJET

View full text Add to dashboard Cite

show abstract

“…Yeruva et al [29] discussed differences between human annotators and machine annotators in sentiment analysis tasks. The authors presented a human-in-the-loop approach to explore human-machine collaboration for sentiment analysis.…”

Section: Sentiment Annotation Techniques In Educationmentioning

confidence: 99%

Sentiment analysis and opinion mining on educational data: A survey

Shaik,

Tao,

Dann

et al. 2023

Preprint

View full text Add to dashboard Cite

Sentiment analysis AKA opinion mining is one of the most widely used NLP applications to identify human intentions from their reviews. In the education sector, opinion mining is used to listen to student opinions and enhance their learning-teaching practices pedagogically. With advancements in sentiment annotation techniques and AI methodologies, student comments can be labelled with their sentiment orientation without much human intervention. In this review article, 1) we consider the role of emotional analysis in education from four levels: document level, sentence level, entity level, and aspect level, 2)sentiment annotation techniques including lexicon-based and corpus-based approaches for unsupervised annotations are explored, 3)the role of AI in sentiment analysis with methodologies like machine learning,

show abstract

“…line, stanza and poem, were instead taken into consideration in PO-EMO and THU-FSPC (Tsinghua University-Fine-grained Sentimental Poetry Corpus) (Chen et al 2019). The only annotation at sentence level, that is an intermediate unit of analysis, is reported by Yeruva et al (2020) on Aeschylus's tragedies; r granularity of classification (from binary classes to wide sets of emotions): for example, Kabithaa has only two labels (positive and negative), while both Kāvi and PERC are based on the Indian concept of Navrasa that distinguishes nine emotions, both positive, such as shaanti (meaning 'peace'), and negative, such as raudra (meaning 'anger'); r perspective (annotation of the emotions as intended by the author or as perceived by the reader): both approaches are covered by the available datasets. It is interesting to notice that during a preliminary annotation of the Iliad, both perspectives were taken into consideration and annotated by two different groups.…”

Section: Related Workmentioning

confidence: 99%

The Sentiment of Latin Poetry. Annotation and Automatic Analysis of the Odes of Horace

Sprugnoli,

Mambrini,

Passarotti

et al. 2023

ijcol

View full text Add to dashboard Cite

During the recent years, an always growing number of linguistic resources and automatic systems for sentiment analysis have been developed covering a wide range of languages. However, research in this field is still not much explored for texts written in Classical languages. Working on such languages means dealing with peculiar textual genres such as philosophical, historical or religious treatises, epic narratives, plays and poems. Poems are particularly suitable for sentiment analysis because they tell us about emotions and passions. In this paper, we describe the creation of the first small gold standard of Latin made of poems written by Horace and manually annotated with emotion polarity, but we also report about the results of a set of automatic classification experiments. In particular, we test both a lexicon-based approach, which uses a Latin polarity lexicon called LatinAffectus, and a zero-shot transfer method. We provide details about the methodology adopted for the annotation of the gold standard, the creation of LatinAffectus, the development of our experiments and we give details about the results and the limitations of the proposed approaches. 01

show abstract

Interpretation of Sentiment Analysis with Human-in-the-Loop

Cited by 7 publications

References 13 publications

Development and Evaluation of Gold Standard Dataset for Sentiment Analysis of Tweets

Development and Evaluation of Gold Standard Dataset for Sentiment Analysis of Tweets

Sentiment analysis and opinion mining on educational data: A survey

The Sentiment of Latin Poetry. Annotation and Automatic Analysis of the Odes of Horace

Contact Info

Product

Resources

About