MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

Abstract

Reconstructing the 3D geometry of the surgical site and detecting instruments within it are important tasks for surgical navigation systems and robotic surgery automation. Traditional approaches treat each problem in isolation and do not account for the intrinsic relationship between segmentation and stereo matching. In this paper, we present a learning-based framework that jointly estimates disparity and binary tool segmentation masks. The core component of our architecture is a shared feature encoder which allows strong interaction between the aforementioned tasks. Experimentally, we train two variants of our network with different capacities and explore different training schemes including both multi-task and single-task learning. Our results show that supervising the segmentation task improves our network's disparity estimation accuracy. We demonstrate a domain adaptation scheme where we supervise the segmentation task with monocular data and achieve domain adaptation of the adjacent disparity task, reducing disparity End-Point-Error and depth mean absolute error by 77.73% and 61.73% respectively compared to the pre-trained baseline model. Our best overall multi-task model, trained with both disparity and segmentation data in subsequent phases, achieves 89.15% mean Intersection-over-Union in RIS and 3.18 millimetre depth mean absolute error in SCARED test sets. Our proposed multi-task architecture is real-time, able to process (1280x1024) stereo input and simultaneously estimate disparity maps and segmentation masks at 22 frames per second. The model code and pre-trained models are made available: https://github.com/dimitrisPs/msdesis

Dense 3D Reconstruction Through Lidar: A New Perspective on Computer-Integrated Surgery

“…Computational efficiency represents a further challenge for real-time perception. A recent trend in this direction is the combination of 3D reconstruction with other surgical computer-vision tasks, such as tool segmentation or tissue tracking, in synergistic (and ideally more efficient) network architectures [38], [39]. Nevertheless, the processing of highresolution raw images to produce 3D point clouds still introduces significant delays.…”

MFF‐Net: Multiscale feature fusion semantic segmentation network for intracranial surgical instruments

“…The detection and segmentation technology of surgical instruments can help doctors quickly locate and recognise the shape and posture of intracranial surgical instruments during craniotomy and can also effectively separate surgical instruments from the surgical scene to avoid accidental injury of surgical instruments to surrounding tissues and organs, thereby improving surgical safety and efficiency. The application of this technology will positively impact the clinical practice of craniotomy, and it is expected to become an important auxiliary tool for craniotomy in the future 5–7 …”

EndoVAE: Generating Endoscopic Images with a Variational Autoencoder

“…The evaluation of the recorded WCE videos usually requires 45-90 minutes, and it is prone to human errors even by experienced clinicians [2]. Aiming to mitigate the risk of such errors, various image-based Clinical Decision Support (CDS) systems have been proposed [3], [4] A significant aspect concerning the performance of CDS systems is their generalization ability. The availability and diversity of training data, directly impact the generalization capability of such systems.…”