Detecting Depression in Dyadic Conversations with Multimodal Narratives and Visualizations

Abstract

Conversations contain a wide spectrum of multimodal information that gives us hints about the emotions and moods of the speaker. In this paper, we developed a system that supports humans to analyze conversations. Our main contribution is the identification of appropriate multimodal features and the integration of such features into verbatim conversation transcripts. We demonstrate the ability of our system to take in a wide range of multimodal information and automatically generated a prediction score for the depression state of the individual. Our experiments showed that this approach yielded better performance than the baseline model. Furthermore, the multimodal narrative approach makes it easy to integrate learnings from other disciplines, such as conversational analysis and psychology. Lastly, this interdisciplinary and automated approach is a step towards emulating how practitioners record the course of treatment as well as emulating how conversational analysts have been analyzing conversations by hand.

An empirical user-study of text-based nonverbal annotation systems for human-human conversations

“…The "Multimodal Narratives for Human" (MONAH) system [6,7], is an automated nonverbal annotation system. The system takes in one video per speaker and runs a pipeline of data processing steps.…”

“…Compared to the Jefferson transcript, a MONAH transcript appears to be easier to read because it does not involve technical symbols. The authors found that the multimodal annotations improved the performance of supervised learning [6,7]. However, there was no mention of whether the annotations improved the users' understanding of the conversation.…”

“…This study seeks to understand how people perceived different text-based multimodal annotation systems to represent videos. In this experiment, we looked at verbatim text, MONAH, MONAH+ [6,7] and Jefferson representations [5]. This study aims to create a cheaper and faster way to translate videos in a manner that captures enough relevant information in the video to interpret the actual scenario in the conversation, by comparing the effectiveness different text-based representations of video.…”

“…The first system is the Jefferson Transcription System [5], as introduced above, it is a verbatim transcript that is richly and manually marked up with symbols representing nonverbal information. The second system is the "Multimodal Narratives for Human" (MONAH) system [6,7], which is an automated system that do not use symbols but words to describe the talk turn. The third system is an add-on from the second system, it is MONAH plus words being enlarged and darkened according to the model's attention on the words (more details later).…”

Safeguarding student privacy in an age of analytics

“…In 2014, Cambridge Analytica harvested data from around 200,000 Facebook accounts to build psychological profiles of 87 million users in an attempt to "target their inner demons" and sway their voting behaviors (Cadwalladr and Graham-Harrison 2018). Advances in machine learning and big data have now enabled the use of heretofore low-risk data, like what one posts publicly on Twitter, to be used to predict likelihood of medical conditions (Schneble et al 2020), contact with a pathogen (Keeling et al 2020), or mental illness (Kim et al 2019). Further, constant surveillance (empowered in part by mobile devices and social media) has enabled governments and corporations to develop social management infrastructures to control citizens', employees', and consumers' behaviors through social credit, targeted influence, or fear of discovery (Kostka 2019;West 2019).…”