Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation

Alapatt, Deepak; Mascagni, Pietro; Vardazaryan, Armine; Garcia, Alain; Okamoto, Nariaki; Mutter, Didier; Marescaux, Jacques; Costamagna, Guido; Dallemagne, Bernard; Padoy, Nicolas

doi:10.48550/arxiv.2112.13815

Cited by 3 publications

(5 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…New open questions arise on how this method may improve model robustness [19], federated learning or semi-/self-supervised learning [20][21][22][23]34]. Furthermore, the proposed method could be applicable to other tasks, such as tool localization and tracking [38], action triplets [39], and video semantic segmentation [40]. Future work will study the value of TRandAugment in these different settings and tasks.…”

Section: Discussionmentioning

confidence: 99%

TRandAugment: temporal random augmentation strategy for surgical activity recognition from videos

et al. 2023

Self Cite

View full text Add to dashboard Cite

Purpose Automatic recognition of surgical activities from intraoperative surgical videos is crucial for developing intelligent support systems for computer-assisted interventions. Current state-of-the-art recognition methods are based on deep learning where data augmentation has shown the potential to improve the generalization of these methods. This has spurred work on automated and simplified augmentation strategies for image classification and object detection on datasets of still images. Extending such augmentation methods to videos is not straightforward, as the temporal dimension needs to be considered. Furthermore, surgical videos pose additional challenges as they are composed of multiple, interconnected, and long-duration activities. Methods This work proposes a new simplified augmentation method, called TRandAugment, specifically designed for long surgical videos, that treats each video as an assemble of temporal segments and applies consistent but random transformations to each segment. The proposed augmentation method is used to train an end-to-end spatiotemporal model consisting of a CNN (ResNet50) followed by a TCN. Results The effectiveness of the proposed method is demonstrated on two surgical video datasets, namely Bypass40 and CATARACTS, and two tasks, surgical phase and step recognition. TRandAugment adds a performance boost of 1–6% over previous state-of-the-art methods, that uses manually designed augmentations. Conclusion This work presents a simplified and automated augmentation method for long surgical videos. The proposed method has been validated on different datasets and tasks indicating the importance of devising temporal augmentation methods for long surgical videos.

show abstract

Section: Discussionmentioning

confidence: 99%

TRandAugment: temporal random augmentation strategy for surgical activity recognition from videos

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…The development of surgical segmentation has been pioneered by the group from Strasbourg, France, with early studies focusing on cataract surgery and minimally invasive gallbladder removal [34]. Although the primordial limitation is the difficulty in obtaining sufficient amounts of datasets of the procedures, an even more time-consuming limitation is the fact that surgeons are needed to annotate these vast repositories of video datasets.…”

Section: Instance/video/surgical Segmentationmentioning

confidence: 99%

“…Although the primordial limitation is the difficulty in obtaining sufficient amounts of datasets of the procedures, an even more time-consuming limitation is the fact that surgeons are needed to annotate these vast repositories of video datasets. In an effort to overcome this limitation, the team from France has proposed the utilization of temporally constrained neural networks (TCNN), which are semi-supervised methods that may facilitate the process of annotation and thus surgical The development of surgical segmentation has been pioneered by the group from Strasbourg, France, with early studies focusing on cataract surgery and minimally invasive gallbladder removal [34]. Although the primordial limitation is the difficulty in obtaining sufficient amounts of datasets of the procedures, an even more time-consuming limitation is the fact that surgeons are needed to annotate these vast repositories of video datasets.…”

Section: Instance/video/surgical Segmentationmentioning

confidence: 99%

“…In an effort to overcome this limitation, the team from France has proposed the utilization of temporally constrained neural networks (TCNN), which are semi-supervised methods that may facilitate the process of annotation and thus surgical segmentation. This is possible by analyzing both temporal and spatial signals via auto encoder networks [34,35]. The goal of this review article is to highlight advances in CV, future review articles will focus on sensors and sensor fusion [36].…”

Section: Instance/video/surgical Segmentationmentioning

confidence: 99%

See 1 more Smart Citation

The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

Gumbs

Grasso²,

Bourdel

et al. 2022

Sensors

View full text Add to dashboard Cite

This is a review focused on advances and current limitations of computer vision (CV) and how CV can help us obtain to more autonomous actions in surgery. It is a follow-up article to one that we previously published in Sensors entitled, “Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?” As opposed to that article that also discussed issues of machine learning, deep learning and natural language processing, this review will delve deeper into the field of CV. Additionally, non-visual forms of data that can aid computerized robots in the performance of more autonomous actions, such as instrument priors and audio haptics, will also be highlighted. Furthermore, the current existential crisis for surgeons, endoscopists and interventional radiologists regarding more autonomy during procedures will be discussed. In summary, this paper will discuss how to harness the power of CV to keep doctors who do interventions in the loop.

show abstract

“…proposed a temporal memory attention network (TMANet) based on a self-attentive mechanism. It integrates long-range temporal relationships over video sequences without requiring exhaustive optical flow prediction Alapatt et al (2021). developed a temporally constrained neural network (TCNN) as a semi-supervised framework for surgical video semantic segmentation.…”

mentioning

confidence: 99%

Advanced Visual SLAM and Image Segmentation Techniques for Augmented Reality

Jiang

Tran

Williams

2022

International Journal of Virtual and Augmented Reality

View full text Add to dashboard Cite

Augmented reality can enhance human perception to experience a virtual-reality intertwined world by computer vision techniques. However, the basic techniques cannot handle complex large-scale scenes, tackle real-time occlusion, and render virtual objects in augmented reality. Therefore, this paper studies potential solutions, such as visual SLAM and image segmentation, that can address these challenges in the augmented reality visualizations. This paper provides a review of advanced visual SLAM and image segmentation techniques for augmented reality. In addition, applications of machine learning techniques for improving augmented reality are presented.

show abstract

Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation

Cited by 3 publications

References 35 publications

TRandAugment: temporal random augmentation strategy for surgical activity recognition from videos

TRandAugment: temporal random augmentation strategy for surgical activity recognition from videos

The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

Advanced Visual SLAM and Image Segmentation Techniques for Augmented Reality

Contact Info

Product

Resources

About