On negative results when using sentiment analysis tools for software engineering research

Abstract

Recent years have seen an increasing attention to social aspects of software engineering, including studies of emotions and sentiments experienced and expressed by the software developers. Most of these studies reuse existing sentiment analysis tools such as SENTISTRENGTH and NLTK. However, these tools have been trained on product reviews and movie reviews and, therefore, their results might not be applicable in the software engineering domain. In this paper we study whether the sentiment analysis tools agree with the sentiment recognized by human evaluators (as reported in an earlier study) as well as with each other. Furthermore, we evaluate the impact of the choice of a sentiment analysis tool on software engineering studies by conducting a simple study of differences in issue resolution times for positive, negative and neutral texts. We repeat the study for seven datasets (issue trackers and STACK OVERFLOW questions) and different sentiment analysis tools and observe that the disagreement between the tools can lead to diverging conclusions. Finally, we perform two replications of previously published studies and observe that the results of those studies cannot be confirmed when a different sentiment analysis tool is used.

A large-scale, in-depth analysis of developers’ personalities in the Apache ecosystem

“…Still, many of these tools have not been designed or trained for handling the technical content typical of the software domain [127]. For instance, Jongeling et al [128] have compared various sentiment analysis tools used in previous studies in software engineering and found that they can disagree with the manual labeling of corpora performed by individuals as well as with each other. Therefore, we advocate caution when drawing conclusions from NLP tools not specifically trained for the specific purpose and lexicon, and we acknowledge this as a potential threat to instrumentation validity.…”

SEntiMoji: an emoji-powered learning approach for sentiment analysis in software engineering

“…However, some researchers noticed unreliable results when directly employing such tools for SE tasks [38,41]. Jongeling et al [38] observed the disagreement among these existing tools on the datasets in SE and found that the results of several SE studies involving these sentiment analysis tools cannot be confirmed when a different tool is used. To investigate the challenges in sentiment analysis in SE, Islam and Zibran [35] applied the most popular SentiStrength to some labeled issue comments extracted from JIRA issue tracking system and conducted an indepth qualitative study to uncover twelve difficulties in identifying the sentiments of SE-related texts by analyzing the misclassified samples.…”

“…Previous studies found that sentiment analysis tools trained on non-technical texts are not adequate for SE tasks [35,38] and a lack of domain-specific knowledge is the main reason [35]. Since then, many studies focused on how to use labeled SE-related texts to train SE-customized sentiment classifiers [8,14,35].…”

“…It aims to identify the affective states and subjective opinions in texts. Many out-of-the-box sentiment analysis tools (e.g., SentiStrength [63]) not designed for SE-related texts have been applied to SE tasks, but recent work has indicated that they cannot produce reliable results on SE tasks [38]. Furthermore, Islam and Zibran [35] applied SentiStrength to an SErelated dataset and found that misunderstanding of domain-specific meanings of words (namely technical jargon in the rest of this paper) accounts for the most misclassifications.…”

Sentiment Classification Using N-Gram Inverse Document Frequency and Automated Machine Learning

“…Because of the poor accuracy of existing sentiment analysis tools trained with general sentiment expressions [4], recent studies have tried to customize such tools with software engineering datasets [5]. However, it is reported that no tool is ready to accurately classify sentences to negative, neutral, or positive, even if tools are specifically customized for certain software engineering tasks [5].…”