Accurate depth estimation of skin surface using a light‐field camera toward dynamic haptic palpation

Abstract

Background Haptic skin palpation with three‐dimensional skin surface reconstruction from in vivo skin images in order to acquire both tactile and visual information has been receiving much attention. However, the depth estimation of skin surface, using a light field camera that creates multiple images with a micro‐lens array, is a difficult problem due to low‐resolution images resulting in erroneous disparity matching. Methods Multiple low‐resolution images decoded from a light field camera have limitations to accurate 3D surface reconstruction needed for haptic palpation. To overcome this, a deep learning method, Generative Adversarial Networks, was employed to generate super‐resolved skin images that preserve surface detail without blurring, and then, accurate skin depth was estimated by taking multiple subsequent steps including lens distortion correction, sub‐pixel shifted image generation using phase shift theorem, cost‐volume building, multi‐label optimization, and hole filling and refinement, which is a new approach for 3D skin surface reconstruction. Results Experimental results of the deep‐learning‐based super‐resolution method demonstrated that the textural detail (wrinkles) of super‐resolved skin images is well preserved, unlike other super‐resolution methods. In addition, the depth maps computed with our proposed algorithm verify that our method can produce more accurate and robust results compared to other state‐of‐the‐art depth map computation methods. Conclusion Herein, we first proposed depth map estimation of skin surfaces using a light field camera and subsequently tested it with several skin images. The experimental results established the superiority of the proposed scheme.

Adaptive Fine Distortion Correction Method for Stereo Images of Skin Acquired with a Mobile Phone

“…It is an objective, non-invasive technique that is used to identify skin surface features from two different perspectives and estimate depths through triangulation. This technique can estimate depth values of less than a millimeter and can thus be effectively utilized for the 3D analysis of the skin surface structure [ 6 , 7 , 8 ]. In this study, we established a non-convergence model using a mobile camera and acquired stereo skin images for a 3D skin analysis.…”

Skin tactile surface restoration using deep learning from a mobile image: An application for virtual skincare

“…For this reason, technology development has actively sought to precisely recover tactile information using optical imaging techniques for skin images acquired by an electronic microscope, a mobile phone camera, or a light-field camera. [5][6][7][8][9][10][11][12][13][14] These technologies are aimed at generating tactile skin surfaces and providing realistic tactile force feedback using a haptic feedback device that enables virtual palpation or virtual skincare through touch, which was not possible in the past. The most important aspect of virtual touch (haptic) interface technology for tele-skincare is the generation of an accurate 3D tactile surface model that is free from light distortions, which often results in an overelaborated surface, especially on the focus spot when reconstructing the 3D surface from a mobile skin image.…”