Recent Advancements in Augmented Reality for Robotic Applications: A Survey

Fu, J.; Rota, Alberto; Li, Shufei; Jianzhuang, Zhao,; Liu, Qingsheng; Iovene, Elisa; Ferrigno, Giancarlo; Momi, Elena De

doi:10.3390/act12080323

Cited by 18 publications

(9 citation statements)

References 141 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This permits determination of precisely the disorder source and required action utilizing patient individual modeling from deep learning databases. Moreover, the association AR-DT allows an important accuracy in the domain of suturing, tying, and placement contrasted to hand operations [155][156][157][158][159].…”

Section: Discussionmentioning

confidence: 99%

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

Razek

2023

Applied Sciences

View full text Add to dashboard Cite

This contribution is part of the objective of diligent universal care that ensures the well-being of a patient. It aims to analyze and propose enriched image-guided procedures for surgical interventions and restricted delivery of implanted drugs in minimally invasive and non-ionizing circumstances. This analysis is supported by a literature review conducted in two ways. The first aims to illustrate the importance of recent research and applications involved in different topics of the subject; this is mainly the case for the introduction’s literature. The second concerns the literature dedicated to having more detailed information in context; this mainly concerns the citations in the different sections of the article. The universal goals of medical treatments are intended to involve the well-being of the patient and allow medical personnel to test new therapies and carry out therapeutic training without risk to the patient. First, the various functionalities involved in these procedures and the concerns of the magnetic resonance imaging technique (MRI) and ultrasound imaging technique (USI), recent contributions to the subject are reviewed. Second, the intervention procedures guided by the image and the implemented actions are analyzed. Third, the components of the fields involved in MRI are examined. Fourth, the MRI control of the treatments, its performance and its compliance are analyzed. Compatibility with MRI via electromagnetic compatibility (EMC) is conferred and demonstrated for an actuation example. Fifth, the extension of the concepts mentioned in the article, in the context of patient comfort and the training of medical staff is proposed. The main contribution of this article is the identification of the different strategic aids needed in healthcare related to image-assisted robotics, non-ionized, minimally invasive and locally restrictive means. Furthermore, it highlights the benefits of using phantoms based on real biological properties of the body, digital twins under human control, artificial intelligence tools and augmented reality-assisted robotics.

show abstract

Section: Discussionmentioning

confidence: 99%

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

Razek

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Devices that automate our everyday lives, as in smart cities reported about in [1], and advancements in augmented and virtual reality, as described in [2,3], create a need for new, convenient input methods. HGR is a particularly intuitive form of human-computer interaction (HCI) because people naturally use their hands to communicate with others, as paper [4] states.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Ultrasonic Hand Gesture Recognition

Fertl,

Nguyen,

Krueger

et al. 2024

Sensors

View full text Add to dashboard Cite

As the number of electronic gadgets in our daily lives is increasing and most of them require some kind of human interaction, this demands innovative, convenient input methods. There are limitations to state-of-the-art (SotA) ultrasound-based hand gesture recognition (HGR) systems in terms of robustness and accuracy. This research presents a novel machine learning (ML)-based end-to-end solution for hand gesture recognition with low-cost micro-electromechanical (MEMS) system ultrasonic transducers. In contrast to prior methods, our ML model processes the raw echo samples directly instead of using pre-processed data. Consequently, the processing flow presented in this work leaves it to the ML model to extract the important information from the echo data. The success of this approach is demonstrated as follows. Four MEMS ultrasonic transducers are placed in three different geometrical arrangements. For each arrangement, different types of ML models are optimized and benchmarked on datasets acquired with the presented custom hardware (HW): convolutional neural networks (CNNs), gated recurrent units (GRUs), long short-term memory (LSTM), vision transformer (ViT), and cross-attention multi-scale vision transformer (CrossViT). The three last-mentioned ML models reached more than 88% accuracy. The most important innovation described in this research paper is that we were able to demonstrate that little pre-processing is necessary to obtain high accuracy in ultrasonic HGR for several arrangements of cost-effective and low-power MEMS ultrasonic transducer arrays. Even the computationally intensive Fourier transform can be omitted. The presented approach is further compared to HGR systems using other sensor types such as vision, WiFi, radar, and state-of-the-art ultrasound-based HGR systems. Direct processing of the sensor signals by a compact model makes ultrasonic hand gesture recognition a true low-cost and power-efficient input method.

show abstract

“…Similarly, Farcas et al developed a novel 3D immersive percutaneous renal access VR training platform, and comparison experiments were conducted to verify its superiority compared to the other two PCNL simulation platforms [11]. Furthermore, AR enables users to visualize specific anatomical structures and obtain real-time intraoperative information regarding the patient's anatomy [12]. It combines various sources of information and presents them simultaneously, enhancing the surgical experience [13].…”

Section: Introductionmentioning

confidence: 99%

Augmented Reality and Human–Robot Collaboration Framework for Percutaneous Nephrolithotomy: System Design, Implementation, and Performance Metrics

Fu,

Pecorella,

Iovene

et al. 2024

IEEE Robot. Automat. Mag.

View full text Add to dashboard Cite

During percutaneous nephrolithotomy (PCNL) operations, the surgeon is required to define the incision point on the patient's back, align the needle to a preplanned path, and perform puncture operations afterward. The procedure is currently performed manually using ultrasound or fluoroscopy imaging for needle orientation, which, however, implies limited accuracy and low reproducibility. This work incorporates augmented reality (AR) visualization with an optical see-through head-mounted display (OST-HMD) and human-robot collaboration (HRC) framework to empower the surgeon's task completion performance. In detail, eye-to-hand calibration, system registration, and hologram model registration are performed to realize visual guidance. A Cartesian impedance controller is used to guide the operator during the needle-puncture task execution. Experiments are conducted to verify the system performance com-pared with conventional manual puncture procedures and a 2D monitor-based visualization interface. The results showed that the proposed framework achieves the lowest median and standard deviation error of 3.17 ± 1.36 mm in translation and 1.2 ± .9° in orientation across all of the experimental groups, respectively. Furthermore, the National Aeronautics and Space Administration task load index (NASA-TLX) user evaluation results indicate that the proposed framework requires the lowest workload score of 42.5 ± 13.7 for task completion compared to other experimental setups. The proposed framework exhibits significant potential for clinical application in the PCNL task, as it enhances the surgeon's perception capability, facilitates collision-free needle insertion path planning, and minimizes errors in task completion.PCNL is a well-established minimally invasive surgery (MIS) procedure for accessing, locating, and removing larger

show abstract

Recent Advancements in Augmented Reality for Robotic Applications: A Survey

Cited by 18 publications

References 141 publications

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

Image-Guided Surgical and Pharmacotherapeutic Routines as Part of Diligent Medical Treatment

End-to-End Ultrasonic Hand Gesture Recognition

Augmented Reality and Human–Robot Collaboration Framework for Percutaneous Nephrolithotomy: System Design, Implementation, and Performance Metrics

Contact Info

Product

Resources

About