Models of Gender Dysphoria Using Social Media Data for Use in Technology-Delivered Interventions: Machine Learning and Natural Language Processing Validation Study

Abstract

Background The optimal treatment for gender dysphoria is medical intervention, but many transgender and nonbinary people face significant treatment barriers when seeking help for gender dysphoria. When untreated, gender dysphoria is associated with depression, anxiety, suicidality, and substance misuse. Technology-delivered interventions for transgender and nonbinary people can be used discretely, safely, and flexibly, thereby reducing treatment barriers and increasing access to psychological interventions to manage distress that accompanies gender dysphoria. Technology-delivered interventions are beginning to incorporate machine learning (ML) and natural language processing (NLP) to automate intervention components and tailor intervention content. A critical step in using ML and NLP in technology-delivered interventions is demonstrating how accurately these methods model clinical constructs. Objective This study aimed to determine the preliminary effectiveness of modeling gender dysphoria with ML and NLP, using transgender and nonbinary people’s social media data. Methods Overall, 6 ML models and 949 NLP-generated independent variables were used to model gender dysphoria from the text data of 1573 Reddit (Reddit Inc) posts created on transgender- and nonbinary-specific web-based forums. After developing a codebook grounded in clinical science, a research team of clinicians and students experienced in working with transgender and nonbinary clients used qualitative content analysis to determine whether gender dysphoria was present in each Reddit post (ie, the dependent variable). NLP (eg, n-grams, Linguistic Inquiry and Word Count, word embedding, sentiment, and transfer learning) was used to transform the linguistic content of each post into predictors for ML algorithms. A k-fold cross-validation was performed. Hyperparameters were tuned with random search. Feature selection was performed to demonstrate the relative importance of each NLP-generated independent variable in predicting gender dysphoria. Misclassified posts were analyzed to improve future modeling of gender dysphoria. Results Results indicated that a supervised ML algorithm (ie, optimized extreme gradient boosting [XGBoost]) modeled gender dysphoria with a high degree of accuracy (0.84), precision (0.83), and speed (1.23 seconds). Of the NLP-generated independent variables, Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) clinical keywords (eg, dysphoria and disorder) were most predictive of gender dysphoria. Misclassifications of gender dysphoria were common in posts that expressed uncertainty, featured a stressful experience unrelated to gender dysphoria, were incorrectly coded, expressed insufficient linguistic markers of gender dysphoria, described past experiences of gender dysphoria, showed evidence of identity exploration, expressed aspects of human sexuality unrelated to gender dysphoria, described socially based gender dysphoria, expressed strong affective or cognitive reactions unrelated to gender dysphoria, or discussed body image. Conclusions Findings suggest that ML- and NLP-based models of gender dysphoria have significant potential to be integrated into technology-delivered interventions. The results contribute to the growing evidence on the importance of incorporating ML and NLP designs in clinical science, especially when studying marginalized populations.

The LGBTQ+ Minority Stress on Social Media (MiSSoM) Dataset: A Labeled Dataset for Natural Language Processing and Machine Learning

“…Using traditional machine learning algorithms and a neural network, we demonstrated that the features of the LGBTQ+ MiSSoM datasets yield excellent performance metrics in predicting composite minority stress as well as individual factors of minority stress. In fact, our expertlyderived features and rigorously generated ground truth labels improved the prediction of composite minority stress (Saha et al 2019) and factors of minority stress (Cascalheira et al 2023b), and even outperformed neural network models (Cascalheira et al 2022(Cascalheira et al , 2023a.…”

“…LGBTQ+ health, the field of artificial intelligence (AI) has yet to embrace its utility (Saha et al 2019;Cascalheira et al 2022Cascalheira et al , 2023b. This failure is, in part, a result of limited access to robust data sets that specifically capture LGBTQ+ minority stress in a manner that is scientifically rigorous and ethically sound.…”

“…Medicalized language used to describe mental and behavioral health disorders may be useful signals given the LGBTQ+ community's health inequities (Goldbach et al 2014;Pachankis et al 2020;Cascalheira et al 2023b). Thus, text from the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) chapters on anxiety, depression, stress disorders, substance use, and gender dysphoria were mined for the most frequent 10 clinical keywords.…”

“…Because LGBTQ+ people often use clinical words to describe their experiences with gender dysphoria on social media (Cascalheira et al 2023b), and considering the pervasiveness of minority stress (Meyer 2003), it is possible that this group uses other jargon-heavy words to describe their experiences with minority stress. Thus, several seminal papers on minority stress (Meyer 1995(Meyer , 2003Balsam et al 2011;Hendricks and Testa 2012) were mined for the most frequent keywords.…”

Developing technology-based interventions for infectious diseases: ethical considerations for young sexual and gender minority people

“…Evidence indicates that TBIs for STIs can significantly improve medication adherence (e.g., pre-exposure prophylaxis) and clinic attendance, significantly reduce transmission risk behaviors, and are perceived as acceptable and feasible ( 1 , 10 ). Among YSGM people, many of whom have limited knowledge of STIs given their age and reduced access to familial or peer supports, TBIs have the added benefit of accessibility outside of discriminatory medical environments ( 11 ). However, substantial gaps in TBIs for STIs and emerging infections remain, such as developing TBIs for subgroups of the YSGM community [e.g., sexual minority female adolescents ( 1 )] and developing TBIs for emerging infections.…”