Using Machine Learning and Thematic Analysis Methods to Evaluate Mental Health Apps Based on User Reviews

Abstract

The proliferation of smartphones has led to an increase in mobile health (mHealth) apps over the years. Thus, it is imperative to evaluate these apps by identifying shortcomings or barriers hampering effective delivery of intended services. In this paper, we evaluate 104 mental health apps on Google Play and App Store by performing sentiment analysis of 88125 user reviews using machine learning (ML), and then conducting thematic analysis on the reviews. We implement and compare the performance of five classifiers using supervised ML algorithms that are widely used for solving classification problems. The best performing classifier, with F1-score of 89.42%, was then used in predicting the sentiment polarity of reviews. Next, we conduct a thematic analysis of positive and negative reviews to identify themes representing various factors affecting the effectiveness of mental health apps positively and negatively. Our results uncover 21 negative themes and 29 positive themes. The negative themes fall under the following categories: usability issues, content issues, ethical issues, customer support issues, and billing issues. Some of the positive themes include aesthetically pleasing interface, app stability, customizability, high-quality content, content variation/diversity, personalized content, privacy and security, and low-subscription cost. Finally, we offer design recommendations on how the identified negative factors can be tackled to improve the effectiveness of mental health apps.

Arabic Sentiment Analysis on Chewing Khat Leaves using Machine Learning and Ensemble Methods

“…Unstructured data from 3 million news articles on Reuters assisted in identifying the 10 major health issues published in news articles from 2007 to 2017. On the contrary, the analysis of user reviews on mobile health applications was prioritized in [8], collecting data from 104 mobile health applications with approximately 88,125 user reviews. The data were categorized based on each comment's functionality (such as usability, content, customer support, and ethics), the polarity concept was divided into three classes, and five machine classifiers were applied.…”

“…Several studies have been conducted, classifying sentiments as positive, negative, or neutral [3,4,6]. More complex sentiment analysis [7][8][9][10], often referred to as fine-grained, classify datasets into five classes, namely very positive, positive, neutral, negative, and very negative. Moreover, aspect-based sentiment analysis [11][12][13][14] classifies datasets by extracting entities from text.…”

CIAA-RepDroid: A Fine-Grained and Probabilistic Reputation Scheme of Android Apps Based on Sentiment Analysis of Reviews

“…They found that Logistic Regression outperforms K-Nearest Neighbors, Naïve Bayes, and Random Forest in terms of precision, accuracy, recall, and F1. Oyebode et al [30] compare performance of five classifiers on 104 Android mental health apps by performing sentiment analysis of a total of 88125 reviews. The best one comes out with F1-score of 89.42% and was therefore exploited to predict polarities helpful to study specific themes impacting satisfaction.…”

An Unsupervised Sentiment Classification Method Based on Multi-Level Fuzzy Computing and Multi-Criteria Fusion

“…There are two kinds of supervised sentiment classification methods, one is based on machine learning, the other is based on deep learning [6,7]. In the supervised sentiment classification methods, the text representation model and classification method are two very important parts.…”

“…is the member function of review r i belonging to negative sentiment category N. We also choose a semi-trapezoid function as the member function of the review r i , which is presented in Eq. (6).…”