Machine learning of language use on Twitter reveals weak and non-specific predictions

Kelley, Seán; Mhaonaigh, Caoimhe Ní; Burke, Louise M.; Whelan, Robert; Gillan, Claire M.

doi:10.1038/s41746-022-00576-y

Cited by 15 publications

(10 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, it is imperative to confirm the robustness and reliability of these structures. It is for this reason that some studies repeatedly interrogate the same factor structure across studies [e.g., AD, CIT, and SW: (30,36,46,68,101,116,120,124)], establishing that the association between dimensions and cognitive measures is replicable [e.g., (30,40)] and that results extend to diagnosed patients (32). While this is vital work, there are risks too in focusing narrowly on a single dimensional structure; specific factors, like disorders, may get reified as novel questionnaires and become difficult to change.…”

Section: Discussionmentioning

confidence: 99%

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Wise¹,

Robinson²,

Gillan³

2023

Biological Psychiatry

Self Cite

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Wise¹,

Robinson²,

Gillan³

2023

Biological Psychiatry

Self Cite

View full text Add to dashboard Cite

“…Many previous depression and suicide detection studies were conducted on social media rather than clinical interviews [ 18 , 19 ]. Social media texts freely generated by users are often nonspecific to suicide or depression [ 22 ]. This study analyzed participants’ responses to the HAMD questions, specifically focusing on a series of depressive symptomatologies.…”

Section: Discussionmentioning

confidence: 99%

Detection of Suicidal Ideation in Clinical Interviews for Depression Using Natural Language Processing and Machine Learning: Cross-Sectional Study

Li,

Chen,

Law

et al. 2023

JMIR Med Inform

View full text Add to dashboard Cite

Background Assessing patients’ suicide risk is challenging, especially among those who deny suicidal ideation. Primary care providers have poor agreement in screening suicide risk. Patients’ speech may provide more objective, language-based clues about their underlying suicidal ideation. Text analysis to detect suicide risk in depression is lacking in the literature. Objective This study aimed to determine whether suicidal ideation can be detected via language features in clinical interviews for depression using natural language processing (NLP) and machine learning (ML). Methods This cross-sectional study recruited 305 participants between October 2020 and May 2022 (mean age 53.0, SD 11.77 years; female: n=176, 57%), of which 197 had lifetime depression and 108 were healthy. This study was part of ongoing research on characterizing depression with a case-control design. In this study, 236 participants were nonsuicidal, while 56 and 13 had low and high suicide risks, respectively. The structured interview guide for the Hamilton Depression Rating Scale (HAMD) was adopted to assess suicide risk and depression severity. Suicide risk was clinician rated based on a suicide-related question (H11). The interviews were transcribed and the words in participants’ verbal responses were translated into psychologically meaningful categories using Linguistic Inquiry and Word Count (LIWC). Results Ordinal logistic regression revealed significant suicide-related language features in participants’ responses to the HAMD questions. Increased use of anger words when talking about work and activities posed the highest suicide risk (odds ratio [OR] 2.91, 95% CI 1.22-8.55; P=.02). Random forest models demonstrated that text analysis of the direct responses to H11 was effective in identifying individuals with high suicide risk (AUC 0.76-0.89; P<.001) and detecting suicide risk in general, including both low and high suicide risk (AUC 0.83-0.92; P<.001). More importantly, suicide risk can be detected with satisfactory performance even without patients’ disclosure of suicidal ideation. Based on the response to the question on hypochondriasis, ML models were trained to identify individuals with high suicide risk (AUC 0.76; P<.001). Conclusions This study examined the perspective of using NLP and ML to analyze the texts from clinical interviews for suicidality detection, which has the potential to provide more accurate and specific markers for suicidal ideation detection. The findings may pave the way for developing high-performance assessment of suicide risk for automated detection, including online chatbot-based interviews for universal screening.

show abstract

“…However, a previous evaluation of the efficacy of using various machine learning algorithms to automatically identify emotions expressed on Twitter found that the highest performing algorithm achieved an accuracy rate of 65% [ 14 ]. Another recent article found that machine learning was not effective in making meaningful predictions about users’ mental health from language use on social media; further, Twitter metadata and language use was not specific to any one mental health condition [ 59 ].…”

Section: Discussionmentioning

confidence: 99%

Ethical and Methodological Considerations of Twitter Data for Public Health Research: Systematic Review

Takats¹,

Kwan²,

Wormer³

et al. 2022

J Med Internet Res

View full text Add to dashboard Cite

Background Much research is being carried out using publicly available Twitter data in the field of public health, but the types of research questions that these data are being used to answer and the extent to which these projects require ethical oversight are not clear. Objective This review describes the current state of public health research using Twitter data in terms of methods and research questions, geographic focus, and ethical considerations including obtaining informed consent from Twitter handlers. Methods We implemented a systematic review, following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, of articles published between January 2006 and October 31, 2019, using Twitter data in secondary analyses for public health research, which were found using standardized search criteria on SocINDEX, PsycINFO, and PubMed. Studies were excluded when using Twitter for primary data collection, such as for study recruitment or as part of a dissemination intervention. Results We identified 367 articles that met eligibility criteria. Infectious disease (n=80, 22%) and substance use (n=66, 18%) were the most common topics for these studies, and sentiment mining (n=227, 62%), surveillance (n=224, 61%), and thematic exploration (n=217, 59%) were the most common methodologies employed. Approximately one-third of articles had a global or worldwide geographic focus; another one-third focused on the United States. The majority (n=222, 60%) of articles used a native Twitter application programming interface, and a significant amount of the remainder (n=102, 28%) used a third-party application programming interface. Only one-third (n=119, 32%) of studies sought ethical approval from an institutional review board, while 17% of them (n=62) included identifying information on Twitter users or tweets and 36% of them (n=131) attempted to anonymize identifiers. Most studies (n=272, 79%) included a discussion on the validity of the measures and reliability of coding (70% for interreliability of human coding and 70% for computer algorithm checks), but less attention was paid to the sampling frame, and what underlying population the sample represented. Conclusions Twitter data may be useful in public health research, given its access to publicly available information. However, studies should exercise greater caution in considering the data sources, accession method, and external validity of the sampling frame. Further, an ethical framework is necessary to help guide future research in this area, especially when individual, identifiable Twitter users and tweets are shared and discussed. Trial Registration PROSPERO CRD42020148170; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=148170

show abstract

Machine learning of language use on Twitter reveals weak and non-specific predictions

Cited by 15 publications

References 84 publications

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Detection of Suicidal Ideation in Clinical Interviews for Depression Using Natural Language Processing and Machine Learning: Cross-Sectional Study

Ethical and Methodological Considerations of Twitter Data for Public Health Research: Systematic Review

Contact Info

Product

Resources

About