MEDIASSIST: medical assistance for intraoperative skill transfer in minimally invasive surgery using augmented reality

Surgical process modelling: a review

“…It is also necessary for bottom-up approaches to have a representation of the sequence of surgery through ontologies or a simple list of phases/steps/activities. At the highest level, we find the heavyweight ontologies, which have been used to represent the detailed context of a SPM study (Burgert et al, 2006;Fischer et al, 2005;Katic et al, 2010;Malarme et al, 2010;Speidel et al, 2008;Sudra et al, 2007). A heavyweight ontology is a lightweight ontology, i.e.…”

“…Studies have focused on the recording of the entire procedure , of the phases (Qi et al, 2006), of the steps (Burgert et al, 2006;Fischer et al, 2005;Lemke et al, 2004), of the activities (Forestier et al, 2012;Meng et al, 2004;Neumuth et al, 2006Neumuth et al, , 2009Neumuth et al, , 2012aNeumuth et al, , 2012bRiffaud et al, 2011) and of the motions (Kragic and Hager, 2003). But efforts have been made in particular on extracting low-level information from the OR: videos (Bhatia et al, 2007;Haro et al, 2012;Klank et al, 2008;Lalys et al, 2012aLalys et al, , 2012bLo et al, 2003;Speidel et al, 2008), audio, position data (Houliston et al, 2011;Katic et al, 2010;Ko et al, 2007;Sudra et al, 2007), hand/tool/surgical staff trajectories Ibbotson et al, 1999;Lin et al, 2006;Miyawaki et al, 2005;Nara et al, 2011;Nomm et al, 2008;Yoshimitsu et al, 2010), information about the presence/absence of surgical tools (Ahmadi et al, 2006;Bouarfa et al, 2010;Padoy et al, 2007) or vital signs (Xiao et al, 2005). Several elements of this low-level information can also be combined (Agarwal et al, 2007;James et al, 2007;Malarme et al, 2010;Padoy et al, , 2010Suzuki et al, 2012;Thiemjarus et al, 2012).…”

“…Signal processing tools have been used for analysing patient vital signs Xiao et al, 2005) or audio recordings (Suzuki et al, 2012). In the case of top-down analysis, description logic has been tested (Burgert et al, 2006;Fischer et al, 2005;Katic et al, 2010;Lemke et al, 2004;Sudra et al, 2007), as well as model instantiation (Jannin et al, 2003), decision tree (Jannin et al, 2007), inference engine (Malarme et al, 2005) or workflow engine (Qi et al, 2008). In the case of bottom-up analysis, graphical probabilistic models have often been used to describe dependencies between observations.…”

Towards “real-time” tool-tissue interaction detection in robotically assisted laparoscopy

“…[1] have shown that it is possible use time series analysis to build an average surgical workflow model and automatically detect in real-time the steps of a surgical procedure by monitoring the change of surgical instruments, whereas [8] have combined eye-gaze detection and tracking of tool changes to detect with a 75% accuracy a specific step of a cholecystectomy. Other works have focused on automatically detecting the surgical instruments in the endoscopic images for the automatic control of a robotically-held endoscope [20][9] [19], or as a first step towards recognition of surgeon's action [16]. However, it has not yet been shown that the knowledge of the instruments' position and motion could be used to learn the surgeon's action and monitor surgical gesture.…”

Content-based processing and analysis of endoscopic images and videos: A survey

“…Another use case is to simulate the effect of a surgical step without actually executing it [115]. Speidel et al [214] and [228] propose an AR system for warning in case of risk situations. The system alerts the surgeon if an instruments comes too close to a risk structure (ductus cysticus or arteria cystica in the case of cholecystectomy).…”