Virtual reality for synergistic surgical training and data generation

Munawar, Adnan; Li, Zhaoshuo; Kunjam, Punit; Nagururu, Nimesh V.; Ding, Andy S.; Kazanzides, Peter; Looi, Thomas; Creighton, Francis X.; Taylor, Russell H.; Unberath, Mathias

doi:10.1080/21681163.2021.1999331

Cited by 17 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We distinguish between these two facets of simulation, the visual and the physical, since many simulation frameworks at this moment focus on one or the other. [98] Python, ROS ✓ Physics-based endoscopic simulator AMBF Drilling Simulator [28] C++, Python, ROS ✓ Physics-based microscopic simulator AMBF-RL [99] Python, ROS ✓ Physics-based endoscopic simulator VisionBlender [97] Blender [80] plug-in ✓ Physics-based monoscopic/stereo simulator SurRol [29] Python ✓ Physics-based endoscopic simulator…”

Section: Simulation Methods For Mismentioning

confidence: 99%

“…For example, fluid dynamics and deformable tissue modeling are more computationally intensive than rigid body simulation, and recent work has focused on accelerating these capabilities to enable real-time intra-operative modeling for in silico surgical trials [95,96]. For in silico training, physics engines have been developed prominently for applications in endoscopy and stereo microscopy [28,45,97], where visible sensors more closely resemble RGB(-D) cameras commonly deployed in general robotics.…”

Section: Physics Enginementioning

confidence: 99%

“…In addition, the simulator supports non-rigid bodies, such as soft tissue, and built-in models of real-world surgical robots. These properties have supported immersive simulation of lateral skull-based surgery for simultaneous training and data generation, suitable for training machine learning models [28].…”

Section: Asynchronous Multi-body Framework (Ambf)mentioning

confidence: 99%

“…Although many of these challenges are common in other surgical domains, they can be especially prominent in MIS [24]. Moving forward, intelligent surgical systems promise to preserve the distinct advantages of MIS while offering to alleviate remaining challenges in its delivery, for example by improving intra-operative image quality [25], providing real-time image analysis [26], actuating robotic imaging devices [18,19,27], and performing skilled motor actions in the surgical setting [28,29].…”

Section: Introductionmentioning

confidence: 99%

“…This approach is already prominent in related problems of general robotic manipulation [41,42] and self-driving [43], each of which contends with challenges specific to the given task, environment, and data modality. In the context of MIS, numerous simulation frameworks have been developed by researchers either with machine learning specifically in mind [28,44,45] or as training tools for clinicians and subsequently adapted for machine learning applications [29,37]. One of the most mature examples is the SimNow software suite, developed for the da Vinci Surgical Robot, which includes training routines for 33 surgical skills [46], as well as complete procedures (Intuitive Surgical Operations, Inc.; Sunnyvale, California).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

et al. 2023

Self Cite

View full text Add to dashboard Cite

To mitigate the challenges of operating through narrow incisions under image guidance, there is a desire to develop intelligent systems that assist decision making and spatial reasoning in minimally invasive surgery (MIS). In this context, machine learning-based systems for interventional image analysis are receiving considerable attention because of their flexibility and the opportunity to provide immediate, informative feedback to clinicians. It is further believed that learning-based image analysis may eventually form the foundation for semior fully automated delivery of surgical treatments. A significant bottleneck in developing such systems is the availability of annotated images with sufficient variability to train generalizable models, particularly the most recently favored deep convolutional neural networks or transformer architectures. A popular alternative to acquiring and manually annotating data from the clinical practice is the simulation of these data from human-based models. Simulation has many advantages, including the avoidance of ethical issues, precisely controlled environments, and the scalability of data collection. Here, we survey recent work that relies on in silico training of learning-based MIS systems, in which data are generated via computational simulation. For each imaging modality, we review available simulation tools in terms of compute requirements, image quality, and usability, as well as their applications for training intelligent systems. We further discuss open challenges for simulation-based development of MIS systems, such as the need for integrated imaging and physical modeling for non-optical modalities, as well as generative patient models not dependent on underlying CT, MRI, or other patient data. In conclusion, as the capabilities of in silico training mature, with respect to sim-to-real transfer, computational efficiency, and degree of control, they are contributing toward the next generation of intelligent surgical systems.

show abstract

Section: Simulation Methods For Mismentioning

confidence: 99%

Section: Physics Enginementioning

confidence: 99%

Section: Asynchronous Multi-body Framework (Ambf)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

Dynamic Tactile Synthetic Tissue: from Soft Robotics to Hybrid Surgical Simulators

Thurner,

Maier,

Kaltenbrunner

et al. 2024

Advanced Intelligent Systems

View full text Add to dashboard Cite

Surgical simulators are valuable educational tools for physicians, enhancing their proficiency and improving patient safety. However, they typically still suffer from a lack of realism as they do not emulate dynamic tissue biomechanics haptically and fail to convincingly mimic real‐time physiological reactions. This study presents a dynamic tactile synthetic tissue, integrating both sensory and actuatory capabilities within a fully soft unit, as a core component for soft robotics and future hybrid surgical simulators utilizing dynamic physical phantoms. The adaptive surface of the tissue replica, actuated via hydraulics, is assessed by an embedded carbon black silicone sensor layer using electrical impedance tomography to determine internally or externally induced deformations. The integrated fluid chambers enable pressure and force measurements. The combination of these principles enables real‐time tissue feedback as well as closed loop operation, allowing optimal interaction with the environment. Based on the concepts of soft robotics, such artificial tissues find broad applicability, demonstrated via a soft gripper and surgical simulation applications including a dynamic, artificial brain phantom as well as a synthetic, beating heart. These advancements pave the way toward enhanced realism in surgical simulators including reliable performance evaluation and bear the potential to transform the future of surgical training methodologies.

show abstract

Research on hybrid synchronization methods in multi‐user collaborative VR simulation medical surgery training system

Gong,

Cheng,

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryThe simulation training system based on VR technology has been applied in clinical medicine due to its immersive interactive experience, economy, and safety. In response to the limitation that most medical surgery training systems can only achieve single user operations, a hybrid synchronous model based multi‐user collaborative framework design is proposed to meet the needs of collaborative training in virtual reality teaching processes. Based on the characteristics of frame synchronization and state synchronization, the synchronization method has been improved. Based on frame synchronization, specific state synchronization is used on interactive events to improve the consistency of operation details and real‐time action. On this basis, a system prototype has been implemented using Unity 3D, and compared with traditional schemes, analysis shows that it has faster response speed for disconnected reconnection and more stable interaction state.

show abstract

Virtual reality for synergistic surgical training and data generation

Cited by 17 publications

References 23 publications

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

Dynamic Tactile Synthetic Tissue: from Soft Robotics to Hybrid Surgical Simulators

Research on hybrid synchronization methods in multi‐user collaborative VR simulation medical surgery training system

Contact Info

Product

Resources

About