Face, content, construct and concurrent validity of dry laboratory exercises for robotic training using a global assessment tool

Abstract

Objectives To evaluate robotic dry laboratory (dry lab) exercises in terms of their face, content, construct and concurrent validities. To evaluate the applicability of the Global Evaluative Assessment of Robotic Skills (GEARS) tool to assess dry lab performance. Materials and Methods Participants were prospectively categorized into two groups: robotic novice (no cases as primary surgeon) and robotic expert (≥30 cases). Participants completed three virtual reality (VR) exercises using the da Vinci Skills Simulator (Intuitive Surgical, Sunnyvale, CA, USA), as well as corresponding dry lab versions of each exercise (Mimic Technologies, Seattle, WA, USA) on the da Vinci Surgical System. Simulator performance was assessed by metrics measured on the simulator. Dry lab performance was blindly video‐evaluated by expert review using the six‐metric GEARS tool. Participants completed a post‐study questionnaire (to evaluate face and content validity). A Wilcoxon non‐parametric test was used to compare performance between groups (construct validity) and Spearman's correlation coefficient was used to assess simulation to dry lab performance (concurrent validity). Results The mean number of robotic cases experienced for novices was 0 and for experts the mean (range) was 200 (30–2000) cases. Expert surgeons found the dry lab exercises both ‘realistic’ (median [range] score 8 [4–10] out of 10) and ‘very useful’ for training of residents (median [range] score 9 [5–10] out of 10). Overall, expert surgeons completed all dry lab tasks more efficiently (P < 0.001) and effectively (GEARS total score P < 0.001) than novices. In addition, experts outperformed novices in each individual GEARS metric (P < 0.001). Finally, in comparing dry lab with simulator performance, there was a moderate correlation overall (r = 0.54, P < 0.001). Most simulator metrics correlated moderately to strongly with corresponding GEARS metrics (r = 0.54, P < 0.001). Conclusions The robotic dry lab exercises in the present study have face, content, construct and concurrent validity with the corresponding VR tasks. Until now, the assessment of dry lab exercises has been limited to basic metrics (i.e. time to completion and error avoidance). For the first time, we have shown it is feasibile to apply a global assessment tool (GEARS) to dry lab training.

A Systematic Review of Virtual Reality Simulators for Robot-assisted Surgery

“…In validation studies, participants were recruited from the following specialties: urology (n = 15 studies) [10][11][12][13]15,16,21,[25][26][27]29,30,32,36,37]; gynecology (n = 1) [19]; urology and general surgery (n = 1) [24]; gynecology and general surgery (n = 1) [38]; urology, gynecology, and general surgery (n = 3) [20,22,31]; urology, gynecology, and cardiothoracic surgery (n = 1) [14]; and urology, otorhinolaryngology, cardiology, thoracic surgery, and gynecology (n = 1) [23]; in six studies, the specialty was not indicated [17,18,28,[33][34][35]. In skills transfer studies, participants were from gynecology (n = 3) [42][43][44]; urology (n = 2) [13,27]; general surgery (n = 1) [40]; and urology and gynecology (n = 1) [41]; in two studies the specialty was not reported [39,45].…”

The role of simulation in neurosurgery

“…Among the benchmarks to assess validity, there are the concepts of face, content, construct, and criterion [36]. Face validity or realism describes the extent to which the test simulates the condition in the real world.…”

“…Face validity or realism describes the extent to which the test simulates the condition in the real world. Content validity represents the extent to which the measurement reflects the attributes it purports to measure [36]. The minimum requirement for simulator approval is construct validity, which describes objectively if the test is measuring the construct it claims to be measuring.…”

“…The minimum requirement for simulator approval is construct validity, which describes objectively if the test is measuring the construct it claims to be measuring. Mainly, it should be able to distinguish the performance between a junior and senior surgeon [31,36]. Criterion validity corresponds to the extent to which an assessment tool correlates with different performance's measurements.…”

Performance of robotic simulated skills tasks is positively associated with clinical robotic surgical performance

“…Despite this rapid surge in use, the development and validation of training methods has failed to keep pace. Several training platforms have undergone validation testing, including inanimate tasks [1][2][3][4][5], virtual reality exercises [6][7][8][9] and ex vivo models [5,9,10]. Validation work in these studies includes demonstration of face (realism of training tool), content (usefulness as training tool), construct (ability to distinguish between different skill levels) and cross-method (correlation across training methods) validity.…”