Towards Effective Automatic Feedback for Simulator Training

Marcano, Laura; Yazidi, Anis; Ferati, Mexhid; Komulainen, Tiina M.

doi:10.3384/ecp17138203

Cited by 3 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our previous studies, simulator training data relative to time (in seconds) and precision (in pixels) of image-guided pick-and-place task performance were recorded from different study populations, including absolute beginners, novice surgeons without specific experience in image-guided simulator training, and expert surgeons with variable experience in image-guided surgery [2,3,5]. These previous studies were aimed at investigating the effects of different camera views, monitor positions and levels of expertise on simulator skill statistics.…”

Section: Resultsmentioning

confidence: 99%

“…If this occurs, especially early in a training sequence, there is either a problem with the simulator task (i.e., the task does not produce adequate performance data that allow discriminating between levels of expertise, which is a problem that needs to be fixed), or the trainee is not a true novice. No more than four cases (1)(2)(3)(4) need to be considered for generating clear performance feedback any novice can understand and take into account to attain an optimal speed-precision trade-off.…”

Section: Towards Expert-based Speed-precision Controlmentioning

confidence: 99%

“…Yet, if the simulator does not provide relevant and truly useful instructional feedback to the user, then instructors need to be present to supervise and tutor the trainee. The way feedback is delivered to the operator, and the amount of feedback he/she has to process and integrate, represents an important challenge in the development of automatic systems for simulator training assistance [2,3]. If relevant information is not delivered effectively, either by drowning essential feedback parameters in a large amount of unnecessary ones, or by not providing truly useful feedback that analysis, a principle of automatic expert benchmarking is brought forward to conceptualize key properties of an automatic procedure that (1) knows the expert surgeon's statistics (in-built benchmarks) relative to task execution time and task precision, (2) detects, and if necessary (3) controls for individual speed-precision strategies by comparing a trainee's performance statistics to the expert's benchmarks and, finally, (3) is able to provide appropriate user feedback when necessary.…”

Section: Introductionmentioning

confidence: 99%

“…How expert benchmark statistics may be exploited for automatic performance control is explained.will help the trainee gain proficiency at the task at hand, then operators will not be able to understand the message given by the system, and the latter will have failed its purpose altogether. Guidelines for user interface design and feedback procedures in simulator training contexts do not yet exist, but they can and should be worked out and tested [2,3].Relevant performance metrics [4][5][6][7][8][9][10] are essential to the development of surgical simulator systems for optimal independent training. The presentation of such metrics to the user, in a way that boosts independent learning by producing a measurable skill improvement, is the most important aspect of an effective training system [11].…”

mentioning

confidence: 99%

“…will help the trainee gain proficiency at the task at hand, then operators will not be able to understand the message given by the system, and the latter will have failed its purpose altogether. Guidelines for user interface design and feedback procedures in simulator training contexts do not yet exist, but they can and should be worked out and tested [2,3].…”

mentioning

confidence: 99%

See 4 more Smart Citations

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Dresp-Langley

2018

Information

View full text Add to dashboard Cite

Simulator training for image-guided surgical interventions would benefit from intelligent systems that detect the evolution of task performance, and take control of individual speed-precision strategies by providing effective automatic performance feedback. At the earliest training stages, novices frequently focus on getting faster at the task. This may, as shown here, compromise the evolution of their precision scores, sometimes irreparably, if it is not controlled for as early as possible. Artificial intelligence could help make sure that a trainee reaches her/his optimal individual speed-accuracy trade-off by monitoring individual performance criteria, detecting critical trends at any given moment in time, and alerting the trainee as early as necessary when to slow down and focus on precision, or when to focus on getting faster. It is suggested that, for effective benchmarking, individual training statistics of novices are compared with the statistics of an expert surgeon. The speed-accuracy functions of novices trained in a large number of experimental sessions reveal differences in individual speed-precision strategies, and clarify why such strategies should be automatically detected and controlled for before further training on specific surgical task models, or clinical models, may be envisaged. How expert benchmark statistics may be exploited for automatic performance control is explained.will help the trainee gain proficiency at the task at hand, then operators will not be able to understand the message given by the system, and the latter will have failed its purpose altogether. Guidelines for user interface design and feedback procedures in simulator training contexts do not yet exist, but they can and should be worked out and tested [2,3].Relevant performance metrics [4][5][6][7][8][9][10] are essential to the development of surgical simulator systems for optimal independent training. The presentation of such metrics to the user, in a way that boosts independent learning by producing a measurable skill improvement, is the most important aspect of an effective training system [11]. Metric-based simulation ensures that training sessions are more than just simulated clinical procedures and gets rid of subjectivity in evaluating skill evolution; there is no ambiguity about the progress of training. Benchmarking individual levels of proficiency on the performance levels of experts in a validated, metric-based simulation system has well-established intrinsic face validity [11], and appears a better approach than benchmarking on performance concepts based on expert consensus, for example. Building expert performance in terms of benchmark metrics into simulator training programs provides a sound basis for automatic skill assessment. Benchmarking ensures that the "pass" level is defined by realistic criteria, set directly by the proficiency levels of individuals who are highly experienced at performing the clinical procedures that simulator training is aimed at preparing novices for [11][12][13][14][15].Whether ...

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Towards Expert-based Speed-precision Controlmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Dresp-Langley

2018

Information

View full text Add to dashboard Cite

show abstract

Who Beats the Expert? Building Precision into Simulators for Surgical Skill Assessment

Dresp-Langley¹

2019

BJSTR

View full text Add to dashboard Cite

Simulator training for image-guided surgical interventions allows tracking task performance in terms of speed and precision of task execution. Simulator tasks are more or less realistic with respect to real surgical tasks, and the lack of clear criteria for learning curves and individual skill assessment if more often than not a problem. Recent research has shown that trainees frequently focus on getting faster at the simulator task, and this strategy bias often compromises the evolution of their precision score. As a consequence, and whatever the degree of surgical realism of the simulator task, the first and most critical criterion for skill evolution should be task precision, not the time of task execution. This short opinion paper argues that individual training statistics of novices from a simulator task should therefore always be compared with the statistics of an expert surgeon from the same task. This implies that benchmark statistics from the expert are made available and an objective criterion, i.e. a parameter measure, for task precision is considered for assessing learning curves of novices.

show abstract

Using the concept of data enclosing tunnel as an online feedback tool for simulator training

Marcano

Yazidi

Manca

et al. 2018

Linköping Electronic Conference Proceedings

Self Cite

View full text Add to dashboard Cite

Feedback is one of the key factors that makes industrial simulator training an effective learning tool. Usually, the trainees receive feedback from the instructor, who guides them through the simulation tasks. However, nowadays the availability of expert instructors is scarce while the training demand is increasing. Therefore, there is a need for new simulator training practices that could allow the trainees to be more independent and decrease the need to rely so often on the instructor. This could be achieved by offering the trainees online automated feedback. This article presents a method for developing a tool meeting those requirements is presented. Simulation data were gathered representing different execution paths of the same scenario. Data were then analyzed and clustered using different clustering techniques. Interestingly, "good" and "bad" performances are shown to be separable using different techniques for clustering multivariate time series. Furthermore, we introduce the concept of enclosing data tunnel representing the trajectory of well-behaving execution paths in a reduced dimensional space. By conditioning the mal-behaving performances to be less than 20 % of the total simulation time inside the tunnel, an accuracy on 68 % was obtained. Being more flexible and allowing the mal-behaving performances to be inside the tunnel for a maximum of 35 % of the total simulation time, the accuracy of the enclosing tunnel was increased to 84 %.

show abstract

Towards Effective Automatic Feedback for Simulator Training

Cited by 3 publications

References 10 publications

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Who Beats the Expert? Building Precision into Simulators for Surgical Skill Assessment

Using the concept of data enclosing tunnel as an online feedback tool for simulator training

Contact Info

Product

Resources

About