Multilingual Conversational Systems to Drive the Collection of Patient-Reported Outcomes and Integration into Clinical Workflows

Abstract

Patient-reported outcomes (PROs) and their use in the clinical workflow can improve cancer survivors’ outcomes and quality of life. However, there are several challenges regarding efficient collection of the patient-reported outcomes and their integration into the clinical workflow. Patient adherence and interoperability are recognized as main barriers. This work implements a cancer-related study which interconnects artificial intelligence (spoken language algorithms, conversational intelligence) and natural sciences (embodied conversational agents) to create an omni-comprehensive system enabling symmetric computer-mediated interaction. Its goal is to collect patient information and integrate it into clinical routine as digital patient resources (the Fast Healthcare Interoperability Resources). To further increase convenience and simplicity of the data collection, a multimodal sensing network is delivered. In this paper, we introduce the main components of the system, including the mHealth application, the Open Health Connect platform, and algorithms to deliver speech enabled 3D embodied conversational agent to interact with the cancer survivors in five different languages. The system integrates cancer patients’ reported information as patient gathered health data into their digital clinical record. The value and impact of the integration will be further evaluated in the clinical study.

Multilingual Framework for Risk Assessment and Symptom Tracking (MRAST)

“…The multimodal sensing network [ 107 ] represents the “brain” of the MRAST framework. It consists of components and end-to-end services to facilitate the symmetric interaction [ 45 ], including the speech recognition engine SPREAD, speech synthesis framework PLATOS, natural language services (including a Rasa-based chatbot), and conversational language generation services, i.e., the embodied virtual agent framework EVA [ 108 ]. Additionally, the framework integrates a symptoms extraction and tracking framework, which includes a depression classification pipeline and a risk assessment component built on top of Symptoma AI [ 47 , 109 ] for extracting clinical cues from free text, assessing risk factors, and returning risk scoring.…”

“…As already highlighted in the introduction, the main disadvantages of ePROMs from a patient’s perspective, and reasons for patients not using ePROMs, can be summarized as (i) ability to use (e.g., physical ability due to health issues), (ii) engagement (e.g., patients do not find them relevant because no symptoms exist), (iii) technical issues and usability (e.g., low technical proficiency), (iv) data security and trust [ 14 , 52 ]. Specifically tackling technical issues and usability, chatbots have been an efficient solution to improve usability and simplify the app functionalities and user experience [ 45 , 59 , 63 , 64 , 65 ]. Namely, chatbots exploit artificial intelligence and natural language processing to interact with patients without human intervention.…”

“…The paper is structured as follows. First, we represent the technology of the MRAST framework, which includes the mHealth patient and clinician app; the MSN, which includes the automatic speech recognition (ASR) SPREAD system [ 45 ]; and application of the Symptoma AI, emphasizing the utilization of Symptoma’s proprietary technology for extracting medical risk factors. The technology’s proficiency has been established in previous studies [ 46 , 47 , 48 , 49 ].…”

Advanced Data Processing of Pancreatic Cancer Data Integrating Ontologies and Machine Learning Techniques to Create Holistic Health Records

“…In this respect, health-related data-processing solutions increasingly focus on exploiting value from primary data (coming from established data sources such as lab results, genomics, and family history) or secondary data (coming from IoMT devices that automatically measure and monitor in real time various medical parameters in the human body). The integration of primary and secondary data has revealed the potential for greater insights for healthcare and health-related decision making [ 24 ]. Even if, for collecting prospective and retrospective clinical data, there already exists a plethora of methods and techniques for automatically capturing such data in batches [ 25 ], this is not the case for the ingestion of streaming data, which has come to the attention of research and development during the last five (5) years [ 26 , 27 ].…”

The Human Factors of AI-Empowered Knowledge Sharing

“…Clearly, the poor quality of reports inhibits (AI-facilitated) knowledge sharing among technicians. CUIs have been shown to be a viable alternative to creating reports on paper or with graphical user interfaces, for example, voice-based CUIs are increasingly prevalent in the healthcare domain to support clinical workfows [32,44]. CUIs in healthcare relieve physicians of the burden of documentation by using a digital scribe [11,43].…”