Surgical Gesture Recognition in Laparoscopic Tasks Based on the Transformer Network and Self-Supervised Learning

Abstract

In this study, we propose a deep learning framework and a self-supervision scheme for video-based surgical gesture recognition. The proposed framework is modular. First, a 3D convolutional network extracts feature vectors from video clips for encoding spatial and short-term temporal features. Second, the feature vectors are fed into a transformer network for capturing long-term temporal dependencies. Two main models are proposed, based on the backbone framework: C3DTrans (supervised) and SSC3DTrans (self-supervised). The dataset consisted of 80 videos from two basic laparoscopic tasks: peg transfer (PT) and knot tying (KT). To examine the potential of self-supervision, the models were trained on 60% and 100% of the annotated dataset. In addition, the best-performing model was evaluated on the JIGSAWS robotic surgery dataset. The best model (C3DTrans) achieves an accuracy of 88.0%, a 95.2% clip level, and 97.5% and 97.9% (gesture level), for PT and KT, respectively. The SSC3DTrans performed similar to C3DTrans when training on 60% of the annotated dataset (about 84% and 93% clip-level accuracies for PT and KT, respectively). The performance of C3DTrans on JIGSAWS was close to 76% accuracy, which was similar to or higher than prior techniques based on a single video stream, no additional video training, and online processing.

Automated Assessment of Simulated Laparoscopic Surgical Skill Performance using Deep Learning

“…3DCNNs have been used successfully to learn volumetric data such as CT and MRI, and there are some reports of using 3DCNN to recognize surgical workflow, and others with some success in detecting cancer and abnormal growths in endoscopy [50][51][52][53]. However, there seems to be a paucity in the literature on applying the qualities of 3DCNN to learn spatiotemporal features together to classify between performance levels with laparoscopic surgical skills.…”

Artificial Intelligence for Personalized Genetics and New Drug Development: Benefits and Cautions

“…Fortunately, the technology at our disposal offers cost-effective solutions to these problems. Machine learning (ML) [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15], artificial intelligence (AI) [16][17][18][19][20][21][22][23][24][25][26][27][28][29] and Big Data [30][31][32][33][34][35] can provide robust and innovative answers to long-standing problems.…”