Iterative multi-path tracking for video and volume segmentation with sparse point supervision

Abstract

Recent machine learning strategies for segmentation tasks have shown great ability when trained on large pixel-wise annotated image datasets. It remains a major challenge however to aggregate such datasets, as the time and monetary cost associated with collecting extensive annotations is extremely high. This is particularly the case for generating precise pixel-wise annotations in video and volumetric image data. To this end, this work presents a novel framework to produce pixel-wise segmentations using minimal supervision. Our method relies on 2D point supervision, whereby a single 2D location within an object of interest is provided on each image of the data. Our method then estimates the object appearance in a semi-supervised fashion by learning object-image-specific features and by using these in a semi-supervised learning framework. Our object model is then used in a graph-based optimization problem that takes into account all provided locations and the image data in order to infer the complete pixel-wise segmentation. In practice, we solve this optimally as a tracking problem using a K-shortest path approach. Both the object model and segmentation are then refined iteratively to further improve the final segmentation. We show that by collecting 2D locations using a gaze tracker, our approach can provide state-of-the-art segmentations on a range of objects and image modalities (video and 3D volumes), and that these can then be used to train supervised machine learning classifiers.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

“…Furthermore the aforementioned approaches are fully-supervised and therefore impose important burden on the collection of representative training datasets. New approaches are needed that can generalize easily to various types of procedures and be trained using weaker information for training, such as image-level tool presence [104], point annotation [122] or scribbles [123].…”

A Positive/Unlabeled Approach for the Segmentation of Medical Sequences using Point-Wise Supervision

“…To validate our method, we evaluate it on the publicly available dataset used in Lejeune et al (2018) 3 . It consists of a variety of video and volumes of different modalities with 2D annotation points for different objects of interest, as well as the associated groundtruth segmentations.…”

“…Unlike traditional graphcut based methods (Boykov et al, 2001) that do so by using both positive and negative samples (i.e., PN learning), our focus is on cases where only positive samples are accessible. With some application-specific solutions previously developed (Vilariño et al, 2007;Khosravan et al, 2017), we follow the line of (Bearman et al, 2016;Lejeune et al, 2017Lejeune et al, , 2018, aiming for agnostic solutions capable of working for different unknown object of interest (shape, appearance, motion, etc. ), as well as different imaging modality (MRI, CT-scan, video, etc.).…”

“…In this work, we further restrict the point-wise annotations to only be available on the object of interest, and not on the background regions of the image. Additionally, point-wise annotations can be provided by man-ual clicks (Ferreira et al, 2012;Bearman et al, 2016), or using a gaze tracker (Vilariño et al, 2007;Khosravan et al, 2017;Lejeune et al, 2017Lejeune et al, , 2018. While this convenience and speed gain comes at the cost of extremely sparse annotations in contrast to full semantic segmentations, methods to generate full segmentations in this setting have shown some promising performances in medical imaging.…”

EasyLabels: weak labels for scene segmentation in laparoscopic videos

“…In contrast, approaches using "scribbles" have recently shown greater accuracy and robustness, as they focus on making the network learn from less but still informative, correct information. Some of these approaches are using points [1,11], scribbles [12,20], bounding boxes [24] or even image labels [23]. However, to the best of our knowledge, none of them has yet been applied to surgical data for segmentation purposes.…”