Real-time in-situ visual feedback of task performance in mixed environments for learning joint psychomotor-cognitive tasks

Abstract

This paper proposes an approach to mixed environment training of manual tasks requiring concurrent use of psychomotor and cognitive skills. To train concurrent use of both skill sets, the learner is provided real-time generated, in-situ presented visual feedback of her performance.This feedback provides reinforcement and correction of psychomotor skills concurrently with guidance in developing cognitive models of the task.The general approach is presented: 1) Sensors placed in the physical environment detect in real-time a learner's manipulation of physical objects. 2) Sensor data is input to models of task performance which output quantitative measures of the learner's performance. 3) Pre-defined rules are applied to transform the learner's performance data into visual feedback presented in realtime and in-situ with the physical objects being manipulated.With guidance from medical education experts, we have applied this approach to a mixed environment for learning clinical breast exams (CBEs). CBE belongs to a class of tasks that require learning multiple cognitive elements and task-specific psychomotor skills. Traditional approaches to learning CBEs and other joint psychomotor-cognitive tasks rely on extensive one-onone training with an expert providing subjective feedback. By integrating real-time visual feedback of learners' quantitatively measured CBE performance, a mixed environment for learning CBEs provides on-demand learning opportunities with more objective, detailed feedback than available with expert observation. The proposed approach applied to learning CBEs was informally evaluated by four expert medical educators and six novice medical students. This evaluation highlights that receiving real-time in-situ visual feedback of their performance provides students an advantage, over traditional approaches to learning CBEs, in developing correct psychomotor and cognitive skills.

Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review

“…The study indicates that it is important to determine relevant key features of the task (as also was suggested by Huegel, Celik, Israr, & O'Malley, 2009;Todorov et al, 1997) and to provide feedback only about these key features, in order to not overwhelm the learner with irrelevant information. In realistic virtual reality systems, abstract augmented concurrent feedback has been reported to be detrimental to manual dexterity training of dental students (Wierinck, Puttemans, Swinnen, & van Steenberghe, 2005) but has been rated to be beneficial by medical educators and students for learning clinical breast exams (Kotranza, Lind, Pugh, & Lok, 2009) and has been shown to be effective for robotic laparoscopy training (Judkins, Oleynikov, & Stergiou, 2006).…”

Virtual Technologies Trends in Education

“…In addition, real-time interaction permits visualizing results instantly, so students are able to take decisions based on these results to reach their learning goals, increasing their learning performance and cognitive skills (Kotranza, Lind, Pugh, & Lok, 2009). But it is also possible to interact collaboratively, so virtual environments also boost interaction and collaboration among students.…”

Augmented reality as a new perspective in Dentistry: development of a complementary tool

“…25 In this way, AR technology can provide synthetic objects for teaching, such as anatomical parts or rare items for science laboratories. Kotranza et al (2009) 26 showed an AR system in clinical medicine that embedded touch sensors in a physical environment, collected sensor data to measure learners' performances and then transformed the performance data into visual feedback.…”