Comparing Emotional Valence Scores of Twitter Messages from Human Coding and Machine Learning Algorithms Among Hispanic and African American Family Caregivers of Persons with Dementia

Abstract

We compared emotional valence scores as determined via machine learning approaches to human-coded scores of direct messages on Twitter from our 2,301 followers during a Twitter-based clinical trial screening for Hispanic and African American family caregivers of persons with dementia. We manually assigned emotional valence scores to 249 randomly selected direct Twitter messages from our followers (N=2,301), then we applied three machine learning sentiment analysis algorithms to extract emotional valence scores for each message and compared their mean scores to the human coding results. The aggregated mean emotional scores from the natural language processing were slightly positive, while the mean score from human coding as a gold standard was negative. Clusters of strongly negative sentiments were observed in followers’ responses to being found non-eligible for the study, indicating a significant need for alternative strategies to provide similar research opportunities to non-eligible family caregivers.