The aims of this study were to test the face, content, and construct validities of a virtual-reality haptic arthroscopy simulator and to validate four assessment hypothesis. The participants in our study were 94 arthroscopists attending an international conference on arthroscopy. The interviewed surgeons had been performing arthroscopies for a mean of 8.71 years (σ = 6.94 years). We explained the operation, functionality, instructions for use, and the exercises provided by the simulator. They performed a trial exercise and then an exercise in which performance was recorded. After having using it, the arthroscopists answered a questionnaire. The simulator was classified as one of the best training methods (over phantoms), and obtained a mark of 7.10 out of 10 as an evaluation tool. The simulator was considered more useful for inexperienced surgeons than for surgeons with experience (mean difference 1.88 out of 10, P value < 0.001). The participants valued the simulator at 8.24 as a tool for learning skills, its fidelity at 7.41, the quality of the platform at 7.54, and the content of the exercises at 7.09. It obtained a global score of 7.82. Of the subjects, 30.8% said they would practise with the simulator more than 6 h per week. Of the surgeons, 89.4% affirmed that they would recommend the simulator to their colleagues. The data gathered support the first three hypotheses, as well as face and content validities. Results show statistically significant differences between experts and novices, thus supporting the construct validity, but studies with a larger sample must be carried out to verify this. We propose concrete solutions and an equation to calculate economy of movement. Analogously, we analyze competence measurements and propose an equation to provide a single measurement that contains them all and that, according to the surgeons' criteria, is as reliable as the judgment of experts observing the performance of an apprentice.

This paper presents an assessment methodology to validate surgical simulators which will help researchers in avoiding most common errors by providing a complete structured guide. The methodology organizes the questions depending on the validities they are related to, helping in objectives' definition and consistent hypothesis formulation. We will define the study depending on its purpose, time course and the study factor assignment, taking into account legal and ethical issues and choosing the population and sample size. If it is an experimental study, we will determine if there exist a control group and the operational definition of variables. We will avoid extraneous variables and make our study blind, establishing the final evaluation procedure, and stipulating actuation and observation protocols. A feasibility study will be performed before executing the pilot and final studies in which we will analyse the data as indicated by the statistical plan, obtaining our results and conclusions. We provide an example, applying the assessment methodology step by step to the evaluation of a virtual reality arthroscopy simulator with haptic feedback. Finally, possible experiments are proposed as well as a conscientious study of different alternatives for the final evaluation procedure, and an extended proposal of surgical competence assessment measures.

No abstract

VR Simulators are a powerful alternative to traditional educational techniques in many domains; and in particular, in surgery. Although they offer new possibilities for learning, training and assessment, they still found difficult to be accepted and integrated into hospitals. In this paper, we explain what we consider the key issues to create successful VR simulators, and we present two methodologies: the guidelines for the simulator design and the evaluation of their validity. Research on VR surgical simulators should be interdisciplinary. It involves medicine, educational psychology, computer science, and engineering. Optimal interdisciplinary communication is difficult, and most projects in surgical simulation are strongly influenced by the engineering perspective, with little or no contributions from the others. This unbalance often leads to a premature end of the project or to simulators which are less practical for surgeons. A design methodology should be used as a guide in the process of creating VR simulators. A thorough description of the problem, the simulator’s role, and an exhaustive task analysis will lead to the identification of the requirements. For the technical implementation, decisions will be taken related to the hardware interface and the interaction that users will have with the virtual world; which will determine collision detection and response algorithms, and the behaviour of the 3D models. In addition to the technical testing, it is necessary to prove the validity of the simulator and design procedures to measure the user performance. We explain a methodology to evaluate the validity (face, content, and criterion-related validity), reliability and transfer of skills from a VR simulator to the real environment in a structured and rigorous way. Following this methodology, an evaluation experiment involving 19 orthopaedic doctors using a VR arthroscopy simulator was carried out. Results prove face and content validities, and inform about the factors and measures that are considered important for arthroscopic surgery. In order to consolidate the research results, we encourage the establishment of an intersectorial consortium with agents from the academic, healthcare and industrial sectors to ensure the long-term sustainability of research lines, additional funding, and to guarantee that simulators, once validated, can be widely available in hospitals. This paper presents a global approach including relevant guidelines and methodologies for designing and evaluating VR simulators. It can provide a solid structure for other researchers when facing those processes and contribute to the successful integration of VR simulators within the educational curriculum.