Sentiment polarity analysis of love letters: Evaluation of TextBlob, Vader, Flair, and Hugging Face transformer

Abstract

Sentiment analysis is the task of computationally identifying and quantifying the emotions and opinions expressed in text. However, existing sentiment analysis tools, while increasingly sophisticated, face challenges when applied to complex and personal domains such as love letters. This study investigates the performance and accuracy of four popular Python libraries for sentiment analysis (TextBlob, Vader, Flair, and Hugging Face Transformer) in determining the polarity and intensity of sentiments in love letters. A corpus of 300 love letters was collected and randomly sampled to provide 500 sentences for analysis. Due to the lack of labelled data, human experts participated in evaluating the quality and accuracy of the sentiment annotations. Inter-rater agreements were computed among four judges across randomly sampled sentence lots in two distinct blind rounds. The results reveal varying degrees of effectiveness and agreement among sentiment analysis tools (TextBlob, Vader, Flair, and Hugging Face) and human judges, with Cohen?s Kappa values showing low to moderate agreement (ranging from 0.09 to 0.77), and each tool demonstrating unique strengths?Vader excelling in sentiment intensity and Flair with Hugging Face better at contextual nuances?in handling the emotional complexity of the texts. The study also highlights limitations and proposes some custom metrics for evaluating sentiment analysis tools in the context of love letters, such as tenderness index, passion quotient, nostalgia score, and others. The findings contribute to the emerging field of sentiment analysis and provide insights for developing natural language models better suited for personal and emotionally charged domains.