Background:Augmented reality (AR) technology that can combine computer-generated images with a real scene has been reported in the medical field recently. We devised the AR system for evaluation of improvements of the body surface, which is important for plastic surgery.Methods:We constructed an AR system that is easy to modify by combining existing devices and free software. We superimposed the 3-dimensional images of the body surface and the bone (obtained from VECTRA H1 and CT) onto the actual surgical field by Moverio BT-200 smart glasses and evaluated improvements of the body surface in 8 cases.Results:In all cases, the 3D image was successfully projected on the surgical field. Improvement of the display method of the 3D image made it easier to distinguish the different shapes in the 3D image and surgical field, making comparison easier. In a patient with fibrous dysplasia, the symmetrized body surface image was useful for confirming improvement of the real body surface. In a patient with complex facial fracture, the simulated bone image was useful as a reference for reduction. In a patient with an osteoma of the forehead, simultaneously displayed images of the body surface and the bone made it easier to understand these positional relationships.Conclusions:This study confirmed that AR technology is helpful for evaluation of the body surface in several clinical applications. Our findings are not only useful for body surface evaluation but also for effective utilization of AR technology in the field of plastic surgery.

No abstract

No abstract

In surgical navigation, to accurately know the position of a surgical instrument in a patient's body is very important. Using transparent smart glasses is very useful for surgical navigation because a surgeon does not need to move his/her line of sight from the operative field. We propose a new application software development method that is able to show a stereoscopic vision of highly precise 3D-CG medical models and surgical instruments using transparent smart glasses for surgical navigation. We used Mixed Reality (MR) which is a concept exceeding Augmented Reality (AR) by using Microsoft HoloLens. In Mixed Reality, persons, places, and objects from our physical and virtual worlds merge together in a blended environment. Unlike competitive models, HoloLens can recognize surrounding with a front-facing cameras and 3D depth sensors. External markers and sensors are not required for surrounding recognition. Once a 3D-CG medical model is placed in a blended environment, it is fixed to the place and does not move on its own. Therefore, we can see a stereoscopic vision of a precise medical model projected into our surrounding such as a holographic human. A holographic human is as if he/she is there, which is a special immersive experience we have never felt before. A holographic human can not only be seen, but also can be moved by user’s hand gestures and interactive manipulation is possible. A holographic human and 3D-CG surgical instrument can be displayed simultaneously in a blended environment. The movement of 3D-CG surgical instruments can be linked with actual surgical instruments in the operation room. In the operation room, the holographic human is superimposed on the actual patient position. Since the positional relationship between the holographic human and surgical instruments is clear because it is overlapping, so it is very useful for surgical navigation. Multiple persons can see one holographic human at the same time using multiple HoloLenses. We developed the holographic human application software for surgical navigation using Unity and Vuforia, which are a development software and a library. A holographic vision of a 3D-CG medical model made from an actual patient’s CT/MRI image data is possible using our application software development method. A user can make the application software within only five minutes by preparing 3D-CG medical model file for instance STL. Therefore, surgeon dentists and clinical staff can make the holographic human content easily by themselves. As a result, the method can be utilized daily for routine medical treatment and education.

Several efforts have been made to use virtual reality (VR) and augmented reality (AR) for medical and dental education and surgical support. The current methods still require users to wear devices such as a head-mounted display (HMD) and smart glasses, which pose challenges in hygiene management and long-term use. Additionally, it is necessary to measure the user’s inter-pupillary distance and to reflect it in the device settings each time to accurately display 3D images. This setting is difficult for daily use. We developed and implemented a novel anatomy education method using a spatial reality display capable of stereoscopic viewing with the naked eye without an HMD or smart glasses. In this study, we developed two new applications: (1) a head and neck anatomy education application, which can display 3D-CG models of the skeleton and blood vessels of the head and neck region using 3D human body data available free of charge from public research institutes, and (2) a DICOM image autostereoscopic viewer, which can automatically convert 2D CT/MRI/CBCT image data into 3D-CG models. In total, 104 students at the School of Dentistry experienced and evaluated the system, and the results suggest its usefulness. A stereoscopic display without a head-mounted display is highly useful and promising for anatomy education.