The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

Abstract

This is a review focused on advances and current limitations of computer vision (CV) and how CV can help us obtain to more autonomous actions in surgery. It is a follow-up article to one that we previously published in Sensors entitled, “Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?” As opposed to that article that also discussed issues of machine learning, deep learning and natural language processing, this review will delve deeper into the field of CV. Additionally, non-visual forms of data that can aid computerized robots in the performance of more autonomous actions, such as instrument priors and audio haptics, will also be highlighted. Furthermore, the current existential crisis for surgeons, endoscopists and interventional radiologists regarding more autonomy during procedures will be discussed. In summary, this paper will discuss how to harness the power of CV to keep doctors who do interventions in the loop.

The challenges of deep learning in artificial intelligence and autonomous actions in surgery: a literature review

“…It can be used in the customization of anti-retroviral and anti-epileptic drugs, sepsis management [4] , and insulin administration in artificial pancreatic systems [13] . It is also used to develop autonomous actions in the interventional fields of surgery, interventional radiology, and endoscopy [14][15][16][17] .…”

“…Sensors, in particular, will play an important role in overcoming challenges to the development and implementation of AI by allowing for more accurate perception and capturing of dynamic and complicated environments via the fusion of multimodal data. The placement of sensors at the tips of instruments creates many challenging issues, starting from size constraints to limitations including the need for repetitive sterilization [14] .…”

“…One of the challenges of sensor development is the special focus on haptics, where the detection of a surgeon's hand tremors is the data input to intraoperative machine processing. However, there is currently a debate as to whether there is a need to invest in haptics since the procedure may ultimately be performed without the need for the clinician to sense and interpret haptic information [14] . This is because surgeons who presently use surgical systems can perform surgery without haptics and rely exclusively on visual cues [14] ; this makes the development of more sophisticated image capturing sensors perhaps more relevant.…”

“…Rather than haptics, investigators are paying more attention to the noise developed by those instruments when in use and are developing sensors to capture these audio data. The upside of this type of sensor is that it could be placed on the proximal end of the guide wire and not the distal part inside the patient, avoiding issues of size and regulatory constraints [14] .…”

Survival Study: International Multicentric Minimally Invasive Liver Resection for Colorectal Liver Metastases (SIMMILR-2)

“…Newer collaborative robots that are handheld or do not use a console may also become useful in the surgeon’s armamentarium against CRLM and may actually permit a safer adoption of more automatic and autonomous actions in the operating room [ 58 , 59 , 60 , 61 ]. Modern-day liver surgeons will need expertise in open, laparoscopic and robotic-assisted techniques to offer the best personalized surgical care to their patients with CRLM.…”