Beyond Rating Scales: With Targeted Evaluation, Language Models are Poised for Psychological Assessment

Abstract

Artificial intelligence-based (AI-based) language analysis has been undergoing a purported “paradigm shift” initiated by new machine learning models, large language models. These models, such as GPT3 or BERT, have led to unprecedented accuracies over most computerized language processing tasks such as web search, automatic machine translation, and question answering, while their chat-based forms like ChatGPT have captured the interest of over a million users. The success of the large language model is mostly attributed to its capability to numerically represent words in their context, long a weakness of popular language assessment techniques that use “bag-of-words” or word count approaches. While many potential applications, from therapy to health information education, are beginning to be studied on the heels of chatGPT’s success, here we suggest these models are already ready to improve how we do psychological assessment. We present the case for AI's paradigm shift to large language models to be the missing piece for a parallel transformation in mental health assessment from the strong reliance on rating scale responses.

A step-by-step method for cultural annotation by LLMs

“…Crucially, this study concludes that there is no need to retrain the model to incorporate new knowledge about the cultural artifacts: “this is possible thanks to the memorization capabilities of GPT-3, which at training time has observed millions of tokens regarding domain-specific knowledge” ( Bongini et al, 2023 ). Let us take an example from another domain of human cultures: GPT-4 performed comparably to well-trained law student annotators in analyzing legal texts, demonstrating again its potential in tasks requiring highly specialized domain expertise ( Savelka, 2023 ; Savelka et al, 2023 ; see also, Fink et al, 2023 , for knowledge extraction from lung cancer report; see Hou and Ji, 2023 , for cell type annotation for RNA-seq analysis; see Kjell et al, 2023 , for mental health assessment; see Dillion et al, 2023 , for moral judgements). In all, GPT appears excellent at zero-shot knowledge intensive tasks (see Yang et al, 2023 , for a review).…”

Designing Incremental Knowledge Enrichment in Generative Pre-trained Transformers

“…Challenges related to bias, misinformation, and data quality have been addressed through comprehensive verification frameworks, incorporating both automated algorithms and human oversight [67,68,69,70,71]. The ongoing development of more sophisticated verification and validation methodologies will be critical for the future of trustworthy and reliable AI applications [57,59,72,67].…”

“…Techniques for automated knowledge verification have been developed to ensure the accuracy and consistency of the information processed by AI models [34,47,56,57]. Validation processes have emphasized the importance of contextual relevance and applicability of the knowledge to specific tasks or domains [58,59,60,61]. Peer review mechanisms and consensus algorithms have been utilized to enhance the trustworthiness of collaboratively curated knowledge bases [62,61,63].…”

A Multimodal Approach to Estimate Large Language Model Improvisational Capabilities

“…The capability of those models to interpret and accurately respond to human emotions and intents proves indispensable in environments that demand high levels of user interaction and emotional engagement [30], [31]. Methods to evaluate the emotional and contextual appropriateness of AI responses have been developed, offering crucial insights into the ethical implications and psychological impact of AI systems on human users [32]. The interaction patterns between humans and these AI models have often followed clinically and socially driven operational guidelines, suggesting the potential of AI to significantly augment human capabilities in both routine and sensitive interactions [33], [34].…”