Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks

Mitri, Daniele Di; Schneider, Ján; Specht, Marcus; Drachsler, Hendrik

doi:10.3390/s19143099

Cited by 24 publications

(23 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Two of these sessions, they had to perform correct CPR, while the reminder two sessions they had to perform incorrect executions not locking their arms and not using their body weight. In fact, from the previous study [8] we noticed it was difficult to obtain the full span of mistakes the learners can perform. Asking the experts to mimic the mistakes was, thus, the most sensible option for obtaining a dataset with a balanced class distribution.…”

Section: Phase 1 -Expert Data Collectionmentioning

confidence: 95%

“…In this study, we aimed at overcoming this knowledge gap and by exploring how multimodal data can be used to support psychomotor skill development by providing real-time feedback. We followed a design-based research approach: the presented study is based on the insights of [8], in which we demonstrated that it is possible to detect common CPR mistakes regarding the quality of the chest compressions (CC) (CC-rate, CC-depth and CC-release). In [8], we have also shown that it is possible to extend the common mistake detection of commercial and validated training tools like the Laerdal ResusciAnne manikin with the CPR tutor.…”

Section: Introductionmentioning

confidence: 99%

“…We followed a design-based research approach: the presented study is based on the insights of [8], in which we demonstrated that it is possible to detect common CPR mistakes regarding the quality of the chest compressions (CC) (CC-rate, CC-depth and CC-release). In [8], we have also shown that it is possible to extend the common mistake detection of commercial and validated training tools like the Laerdal ResusciAnne manikin with the CPR tutor. We were able to detect the correct locking of the arms while doing CPR and the correct use of the body weight when performing the CCs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Real-Time Multimodal Feedback with the CPR Tutor

Mitri

Schneider

Trebing

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

We developed the CPR Tutor, a real-time multimodal feedback system for cardiopulmonary resuscitation (CPR) training. The CPR Tutor detects mistakes using recurrent neural networks for real-time time-series classification. From a multimodal data stream consisting of kinematic and electromyographic data, the CPR Tutor system automatically detects the chest compressions, which are then classified and assessed according to five performance indicators. Based on this assessment, the CPR Tutor provides audio feedback to correct the most critical mistakes and improve the CPR performance. To test the validity of the CPR Tutor, we first collected the data corpus from 10 experts used for model training. Hence, to test the impact of the feedback functionality, we ran a user study involving 10 participants. The CPR Tutor pushes forward the current state of the art of real-time multimodal tutors by providing: 1) an architecture design, 2) a methodological approach to design multimodal feedback and 3) a field study on real-time feedback for CPR training.

show abstract

Section: Phase 1 -Expert Data Collectionmentioning

confidence: 95%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Real-Time Multimodal Feedback with the CPR Tutor

Mitri

Schneider

Trebing

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Di Mitri et al investigate to what extent multimodal data could be used to detect mistakes during Cardiopulmonary Resuscitation (CPR) training [5]. They complement the Laerdal QCPR ResusciAnne mannequin with the Multimodal Tutor for CPR, a multi-sensor system consisting of a Microsoft Kinect for tracking body position and a Myo armband for collecting electromyogram information.…”

Section: Contributionsmentioning

confidence: 99%

Advanced Sensors Technology in Education

Crespo

Burgos

2019

Sensors

View full text Add to dashboard Cite

The topic presented will show how different kinds of sensors can help to improve our skills in learning environments. When we open the mind and let it take the control to be creative, we can think how a martial art would be improved with registered sensors, or how a person may dance with machines to improve their technique, or how you may improve your soccer kick for a penalties round. The use of sensors seems easy to imagine in these examples, but their use is not limited to these types of learning environments. Using depth cameras to detect patterns in oral presentations, or improving the assessment of agility through low cost-sensors with multimodal learning analytics, or using computing devices as sensors to measure their impact on primary and secondary students’ performances are the focus of this study as well. We hope readers will find original ideas that allow them to improve and advance in their own researches.

show abstract

“…Data from smartphones have been used to predict human activities such as movement [ 10 , 11 ] and fall detection [ 12 ]. Another external sensor used for training psychomotor skills is Microsoft Kinect [ 13 , 14 , 15 ]. The Kinect is an external infrared sensor that is used to track skeletal points of the human body and their movements, which can also be useful in the case of table tennis.…”

Section: Introductionmentioning

confidence: 99%

Table Tennis Tutor: Forehand Strokes Classification Based on Multimodal Data and Neural Networks

Sanusi

Mitri

Limbu

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

Beginner table-tennis players require constant real-time feedback while learning the fundamental techniques. However, due to various constraints such as the mentor’s inability to be around all the time, expensive sensors and equipment for sports training, beginners are unable to get the immediate real-time feedback they need during training. Sensors have been widely used to train beginners and novices for various skills development, including psychomotor skills. Sensors enable the collection of multimodal data which can be utilised with machine learning to classify training mistakes, give feedback, and further improve the learning outcomes. In this paper, we introduce the Table Tennis Tutor (T3), a multi-sensor system consisting of a smartphone device with its built-in sensors for collecting motion data and a Microsoft Kinect for tracking body position. We focused on the forehand stroke mistake detection. We collected a dataset recording an experienced table tennis player performing 260 short forehand strokes (correct) and mimicking 250 long forehand strokes (mistake). We analysed and annotated the multimodal data for training a recurrent neural network that classifies correct and incorrect strokes. To investigate the accuracy level of the aforementioned sensors, three combinations were validated in this study: smartphone sensors only, the Kinect only, and both devices combined. The results of the study show that smartphone sensors alone perform sub-par than the Kinect, but similar with better precision together with the Kinect. To further strengthen T3’s potential for training, an expert interview session was held virtually with a table tennis coach to investigate the coach’s perception of having a real-time feedback system to assist beginners during training sessions. The outcome of the interview shows positive expectations and provided more inputs that can be beneficial for the future implementations of the T3.

show abstract

Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks

Cited by 24 publications

References 35 publications

Real-Time Multimodal Feedback with the CPR Tutor

Real-Time Multimodal Feedback with the CPR Tutor

Advanced Sensors Technology in Education

Table Tennis Tutor: Forehand Strokes Classification Based on Multimodal Data and Neural Networks

Contact Info

Product

Resources

About