No abstract

Intro Thiel embalmed human cadavers are considered as high‐fidelity simulator for several surgical procedures, including Chest Tube Insertion (CTI). In proficiency‐based education, targeted feedback is essential in order to improve learners’ technical skills. Assessments must be validated prior to their use. Validity can be established using Messick’s framework, which collects evidence from five sources. The ACTION tool is a new assessment tool, and consists of a procedure‐specific Rating Scale and an error checklist. It has been developed by a Delphi panel, which ensures content evidence. This study evaluates the feasibility of a validation study for the ACTION tool on Thiel embalmed human cadavers. Methods A convenience sample of three medical undergraduates and one PhD candidate was recruited. All provided informed consent and demographic information. Each participant was scored and recorded while performing a CTI. Each recording was later scored by three raters. Rater agreement was evaluated by calculating a two‐way mixed effects, absolute agreement, single unit of measurement intraclass correlation coefficient (ICC) model. Internal consistency of the assessment was evaluated using Cronbach Alfa. Feasibility was evaluated using structured questions regarding recruitment and data outcomes. Results Recruitment capability and data collection procedures were evaluated as being feasible and suitable for the target study participants. All necessary resources are available and data analysis can be carried out. Preliminary results for the procedural steps indicate an ICC of 0.62, indicating moderate rater agreement, and a Cronbach Alfa of 0.94, indicating excellent internal consistency. For the errors, an ICC of 0.63, and a Cronbach Alfa of 0.68 was observed. Conclusion Validation of the ACTION tool following Messick’s framework is feasible. A rater training session can increase reliability. After validation, the assessment tool can be used to evaluate learners’ proficiency in CTI on Thiel embalmed human cadavers, prior to patient contact.

Purpose of the studyTo collect validity evidence for the chest tube insertion (CTI) test mode on the medical simulation application Touch Surgery. This was done by using Messick’s contemporary framework.MethodsNovice, intermediate and experienced participants provided informed consent and demographic information. After familiarisation with the application, they completed the CTI test mode. Validity evidence was collected from four sources: content, response process, relation to other variables and consequences. A post-study questionnaire with 5-point Likert scales assessed the perceived realism, relevance and utility of the assessment. Mean scores of the three groups were compared.ResultsA total of 25 novices, 11 intermediates and 19 experienced participants were recruited. Content evidence was collected by an expert in CTI and was based on published literature and guidelines. All users were familiarised with the application, and received standardised instructions throughout the test. Most users rated the simulation as realistic and suitable to assess cognitive skills. Novices received significantly lower (55.9±7.5) test mode scores than intermediates (80.6±4.4) (p<0.001) and experienced participants (82.3±5.3) (p<0.001). There was no significant difference in score between intermediate and experienced participants (p=0.75). Consequences evidence was provided by establishing a pass/fail score of 71% using the contrasting groups method, which resulted in one observed false positive and no false negatives.ConclusionA robust validity argument was constructed for the CTI test mode, which can be implemented in surgical curricula to assess learners’ cognitive skills prior to hands-on simulation practice.