Nextmed: Automatic Imaging Segmentation, 3D Reconstruction, and 3D Model Visualization Platform Using Augmented and Virtual Reality

Abstract

The visualization of medical images with advanced techniques, such as augmented reality and virtual reality, represent a breakthrough for medical professionals. In contrast to more traditional visualization tools lacking 3D capabilities, these systems use the three available dimensions. To visualize medical images in 3D, the anatomical areas of interest must be segmented. Currently, manual segmentation, which is the most commonly used technique, and semi-automatic approaches can be time consuming because a doctor is required, making segmentation for each individual case unfeasible. Using new technologies, such as computer vision and artificial intelligence for segmentation algorithms and augmented and virtual reality for visualization techniques implementation, we designed a complete platform to solve this problem and allow medical professionals to work more frequently with anatomical 3D models obtained from medical imaging. As a result, the Nextmed project, due to the different implemented software applications, permits the importation of digital imaging and communication on medicine (dicom) images on a secure cloud platform and the automatic segmentation of certain anatomical structures with new algorithms that improve upon the current research results. A 3D mesh of the segmented structure is then automatically generated that can be printed in 3D or visualized using both augmented and virtual reality, with the designed software systems. The Nextmed project is unique, as it covers the whole process from uploading dicom images to automatic segmentation, 3D reconstruction, 3D visualization, and manipulation using augmented and virtual reality. There are many researches about application of augmented and virtual reality for medical image 3D visualization; however, they are not automated platforms. Although some other anatomical structures can be studied, we focused on one case: a lung study. Analyzing the application of the platform to more than 1000 dicom images and studying the results with medical specialists, we concluded that the installation of this system in hospitals would provide a considerable improvement as a tool for medical image visualization.

Marching Cubes and Histogram Pyramids for 3D Medical Visualization

“…Texture mapping technique constructs 3D object by interpreting the spatial relationships of 2D binary images and generate 3D visualization of perspective-mapped from each image layer [4]. Whereas, semi-automatic image segmentation allows the doctor to make the segmentation of subject by the area of interest [5]. Therefore, the major drawback of these methods is the computational cost of real time resampling for making texture in 3D object reconstruction process [4], and infeasibility for each individual segmentation [5] (see Figure 1b and 1c).…”

“…Whereas, semi-automatic image segmentation allows the doctor to make the segmentation of subject by the area of interest [5]. Therefore, the major drawback of these methods is the computational cost of real time resampling for making texture in 3D object reconstruction process [4], and infeasibility for each individual segmentation [5] (see Figure 1b and 1c). The alternative choice for 3D volumetric rendering is marching cubes.…”

Marching Cubes and Histogram Pyramids for 3D Medical Visualization

“…Texture mapping technique constructs 3D object by interpreting the spatial relationships of 2D binary images and generates 3D visualization of perspective-mapped from each image layer [ 4 ]. Whereas, semi-automatic image segmentation allows the doctor to make the segmentation of subject by the area of interest [ 5 ]. Therefore, the major drawback of these methods is the computational cost of real time resampling for making texture in 3D object reconstruction process [ 4 ], and infeasibility for each individual segmentation [ 5 ] (see Figure 1 b,c).…”

New methods and technologies for enhancing usability and accessibility of educational data

“…This special issue focuses on how to improve universal access to educational data, with emphasis on (a) new technologies and associated data in educational contexts: artificial intelligence systems [70], robotics [71][72][73], augmented [74][75][76] and virtual reality (VR) [77][78][79][80][81], and educational data integration and management [82]; (b) the role of data in the digital transformation and future of higher education: Personal Learning Environments (PLE) [83,84], mobile PLE [85,86], stealth assessment [87], technology-supported collaboration and teamwork in educational environments [88], and student's engagement and interactions [89,90]; (c) user and case studies on ICTs in education [91,92]; (d) educational data in serious games and gamification: gamification design [93][94][95][96], serious game mechanics for education [97,98], ubiquitous/pervasive gaming [99], and game-based learning and teaching programming [100,101]; and (e) educational data visualization and data mining [102]: learning analytics [103], knowledge discovery [104], user experience [105,106], social impact [107], good practices [108], and accessibility …”