Explaining a model predicting quality of surgical practice: a first presentation to and review by clinical experts

Abstract

Surgical Data Science (SDS) is an emerging research domain offering data-driven answers to challenges encountered by clinicians during training and practice. We previously developed a framework to assess quality of practice based on two aspects: exposure of the surgical scene (ESS ) and the surgeon's profile of practice (SPP ). Here, we wished to investigate the clinical relevance of the parameters learned by this model by 1) interpreting these parameters and identifying associated representative video samples, 2) presenting this information to surgeons in the form of a video-enhanced questionnaire. To our knowledge, this is the first approach in the field of SDS for laparoscopy linking the choices made by a machine learning model predicting surgical quality to clinical expertise.Method. Spatial features and quality of practice scores extracted from labeled and segmented frames in 30 laparoscopic videos were used to predict the ESS and the SPP. The relationships between the inputs and outputs of the model were then analyzed and translated into meaningful sentences (statements, e.g. "To optimize the ESS, it is very important to correctly handle the spleen"). Representative video clips illustrating these statements were semiautomatically identified. Eleven statements and video clips were used in a sur-

Essential properties and explanation effectiveness of explainable artificial intelligence in healthcare: A systematic review

“…In five studies, the aim was to develop an explanatory model to assist healthcare providers in diagnosing a patient's disease [ [29] , [30] , [31] , [32] , [33] ]. The aim of the remaining study was to examine the explainability of the model in order to identify variables that have an important influence on the prediction results [ 34 ]. Four of the six studies utilized publicly available clinical data [ [29] , [30] , [31] , [32] ], and the other two studies utilized electronic health record (EHR) data [ 33 , 34 ].…”

“…The aim of the remaining study was to examine the explainability of the model in order to identify variables that have an important influence on the prediction results [ 34 ]. Four of the six studies utilized publicly available clinical data [ [29] , [30] , [31] , [32] ], and the other two studies utilized electronic health record (EHR) data [ 33 , 34 ]. The studies mainly used image and multimedia data, such as computed tomography (CT) scans [ 33 ], pathogen images [ 32 ], electrocardiogram (ECG) [ 29 , 31 ], ultrasound videos [ 30 ], and operation videos [ 34 ].…”

“…Various types of AI algorithms, including the rule-based algorithm [ 29 ], K-Nearest Neighbors (KNN) [ 31 ], Convolutional Neural Network (CNN) [ [30] , [31] , [32] ], conventional image processing algorithm [ 33 ], and Support Vector Machine (SVM) [ 34 ] were used in the studies. The performance of the algorithms was presented using accuracy, balanced accuracy, sensitivity or recall, specificity, Positive Predictive Value (PPV) or precision, Negative Predictive Value (NPV), F1-score, Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), Matthew's Correlation Coefficient (MCC), and error rate.…”

“…The studies adopted the following explanation approaches: the Permutation Feature Importance (PFI) [ 31 ], Local Interpretable Model-agnostic Explanation (LIME) [ 31 ], SHappley Additive exPlanation (SHAP) [ 31 ], faster Region with Convolutional Neural Network (R-CNN) [ 33 ], Class Activation Map (CAM) [ 30 ], Cumulative Fuzzy Class Membership Criterion (CFCMC) [ 32 ], pseudo-coloring methodology [ 29 ], and value permutation and feature-object semantics [ 34 ]. Brief definitions of these explanation approaches are included in Appendix B.…”

“…All six studies included an evaluation of the explainability primarily for physicians as the target AI users ( Table 3 ) [ [30] , [31] , [32] , [33] , [34] ]. Only one study included comments from patients, individuals affected by AI-based decisions , by presenting the explanations for the XAI model [ 29 ].…”