Assessment study of insight ARTHRO VR ® arthroscopy virtual training simulator: face, content, and construct validities

Abstract

The aims of this study were to test the face, content, and construct validities of a virtual-reality haptic arthroscopy simulator and to validate four assessment hypothesis. The participants in our study were 94 arthroscopists attending an international conference on arthroscopy. The interviewed surgeons had been performing arthroscopies for a mean of 8.71 years (σ = 6.94 years). We explained the operation, functionality, instructions for use, and the exercises provided by the simulator. They performed a trial exercise and then an exercise in which performance was recorded. After having using it, the arthroscopists answered a questionnaire. The simulator was classified as one of the best training methods (over phantoms), and obtained a mark of 7.10 out of 10 as an evaluation tool. The simulator was considered more useful for inexperienced surgeons than for surgeons with experience (mean difference 1.88 out of 10, P value < 0.001). The participants valued the simulator at 8.24 as a tool for learning skills, its fidelity at 7.41, the quality of the platform at 7.54, and the content of the exercises at 7.09. It obtained a global score of 7.82. Of the subjects, 30.8% said they would practise with the simulator more than 6 h per week. Of the surgeons, 89.4% affirmed that they would recommend the simulator to their colleagues. The data gathered support the first three hypotheses, as well as face and content validities. Results show statistically significant differences between experts and novices, thus supporting the construct validity, but studies with a larger sample must be carried out to verify this. We propose concrete solutions and an equation to calculate economy of movement. Analogously, we analyze competence measurements and propose an equation to provide a single measurement that contains them all and that, according to the surgeons' criteria, is as reliable as the judgment of experts observing the performance of an apprentice.

A new assessment methodology for virtual reality surgical simulators

“…We have been able to prove face and content validities and we have found evidence for supporting construct validity of the arthroscopy simulator. 25 During the study of concurrent and predictive validities, it will be interesting to analyse the transfer effectiveness ratio, that is, how many hours on a simulator save how many hours in real work. The execution of the experiment proposed as Category A, with the Integrated assessment procedure for the final evaluation could serve for this purpose.…”

“…25 Our measure, according to surgeons' criteria is as reliable as the judgement of experts observing the performance of an apprentice.…”

Virtual and Augmented Reality in Medical Education

“…Both Arthro MENTOR™ (3D Systems, formerly Simbionix) which was the evolution of ARTHRO-VR ® (GMV) and ArthroSIM have displayed face, content, and construct validity [53][54][55][56]. ArthroS™ (Virtamed) has demonstrated greater face validity than Arthro MENTOR™ and ArthroSIM in a head-to-head comparison study [54].…”

Augmented Virtuality for Arthroscopic Knee Surgery

“…A user study was not performed to validate their system as a training tool. In a similar way, Bayona et al [4] developed a shoulder arthroscopy training simulator with force feedback. Their interface included a virtual external rendering which showed the viewing cone of the arthroscope, but without live images.…”