Speech-Based Activity Recognition for Trauma Resuscitation

Abstract

We present a speech-based approach to recognize team activities in the context of trauma resuscitation. We first analyzed the audio recordings of trauma resuscitations in terms of activity frequency, noise-level, and activity-related keyword frequency to determine the dataset characteristics. We next evaluated different audio-preprocessing parameters (spectral feature types and audio channels) to find the optimal configuration. We then introduced a novel neural network to recognize the trauma activities using a modified VGG network that extracts features from the audio input. The output of the modified VGG network is combined with the output of a network that takes keyword text as input, and the combination is used to generate activity labels. We compared our system with several baselines and performed a detailed analysis of the performance results for specific activities. Our results show that our proposed architecture that uses Melspectrum spectral coefficients features with a stereo channel and activity-specific frequent keywords achieve the highest accuracy and average F1-score.

Digitalization in orthopaedics: a narrative review

“…Several AI algorithms can be used to detect arterial injuries using specific patients` parameters ( 77 ). Audio analysis of spoken words in the resuscitation room can be used for data collection and categorization of the resuscitation phases (e.g., arrival of patient, primary survey, secondary survey) ( 78 ). Machine-learning tools and artificial neural networks (ANN) have been used to develop several systems like smartphone applications and ensemble classifiers as decision tools for the prediction of hemorrhage or need for blood transfusions, including mass transfusion protocols ( 72 , 79 , 80 ).…”

Multi-dimensional task recognition for human-robot teaming: literature review

“…The transcribed words are encoded into n − dimensional vectors [e.g., GloVe vector embeddings ( Pennington et al, 2014 )] to be used as inputs for detecting speech-reliant tasks ( Gu et al, 2019 ). Representative keywords that are spoken more frequently can be used for detecting tasks ( Abdulbaqi et al, 2020 ). Keywords can be detected for every utterance automatically using word-spotting ( Tsai and Hao, 2019 ; Gao et al, 2020 ).…”

“…Identifying keywords for each task is non-trivial and requires considerable human effort. Both transcript and keywords metrics’ sensitivity is indeterminate, but is hypothesized to be medium ( Gu et al, 2019 ; Abdulbaqi et al, 2020 ). The metrics conform with suitability, provided the speech audio is obtained using a wearable microphone that minimize extraneous ambient noise.…”

“…Little research exists on speech-reliant task recognition ( Jagannath et al, 2018 ), with the existing algorithms being based on deep learning (e.g., Gu et al, 2019 ; Abdulbaqi et al, 2020 ). A text-based task recognition algorithm employed a verbal transcript derived from a medical team’s communication as input to predict ten trauma resuscitation tasks ( Gu et al, 2019 ).…”

“…Visual task examples include identifying different objects, and reading (e.g., Bulling et al, 2010 ; Ishimaru et al, 2017 ; Srivastava et al, 2018 ). Speech-reliant tasks are voice articulation dependent, such as communicating over the radio (e.g., Dash et al, 2019 ; Abdulbaqi et al, 2020 ), while auditory tasks are acoustic events in the environment, such as an important announcement or emergency sounds (e.g., Stork et al, 2012 ; Hershey et al, 2017 ; Laput et al, 2018 ). Most existing task recognition approaches focus primarily on detecting physical gross motor and fine-grained motor tasks.…”