Plenoptic cameras in surgical robotics: Calibration, registration, and evaluation

Shademan, Azad; Decker, Ryan; Opfermann, Justin D.; Léonard, Simon; Kim, Peter C.W.; Krieger, Axel

doi:10.1109/icra.2016.7487197

Cited by 22 publications

(23 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, high-speed and large-scale 3D functional imaging of neuronal activity has been demonstrated [7]. Furthermore, first studies for surgical robotics [17], endoscopic application [18] and blood-flow visualization [19] have been performed.The key component of standard plenoptic cameras is a microlens array inserted in the native image plane, that reproduces repeated images of the main camera lens on the sensor behind it [1,15]. This enables reconstruction of light paths, employed, in post-processing, for refocusing different planes, changing point of view and extending depth of field (DOF) within the acquired image.…”

mentioning

confidence: 99%

Diffraction-Limited Plenoptic Imaging with Correlated Light

et al. 2017

View full text Add to dashboard Cite

Traditional optical imaging faces an unavoidable trade-off between resolution and depth of field (DOF). To increase resolution, high numerical apertures (NA) are needed, but the associated large angular uncertainty results in a limited range of depths that can be put in sharp focus. Plenoptic imaging was introduced a few years ago to remedy this trade off. To this aim, plenoptic imaging reconstructs the path of light rays from the lens to the sensor. However, the improvement offered by standard plenoptic imaging is practical and not fundamental: the increased DOF leads to a proportional reduction of the resolution well above the diffraction limit imposed by the lens NA. In this paper, we demonstrate that correlation measurements enable pushing plenoptic imaging to its fundamental limits of both resolution and DOF. Namely, we demonstrate to maintain the imaging resolution at the diffraction limit while increasing the depth of field by a factor of 7. Our results represent the theoretical and experimental basis for the effective development of the promising applications of plenoptic imaging.Plenoptic imaging (PI) is a novel optical method for recording visual information [1]. Its peculiarity is the ability to record both position and propagation direction of light in a single exposure. PI is currently employed in the most diverse applications, from stereoscopy [1][2][3], to microscopy [4][5][6][7], particle image velocimetry [8], particle tracking and sizing [9], wavefront sensing [10][11][12][13] and photography, where it currently enables digital cameras with refocusing capabilities [14,15]. The capability of PI to simultaneously acquire multiple-perspective 2D images brings it among the fastest and most promising methods for 3D imaging with the available technologies [16]. Indeed, high-speed and large-scale 3D functional imaging of neuronal activity has been demonstrated [7]. Furthermore, first studies for surgical robotics [17], endoscopic application [18] and blood-flow visualization [19] have been performed.The key component of standard plenoptic cameras is a microlens array inserted in the native image plane, that reproduces repeated images of the main camera lens on the sensor behind it [1,15]. This enables reconstruction of light paths, employed, in post-processing, for refocusing different planes, changing point of view and extending depth of field (DOF) within the acquired image. However, a fundamental trade-off between spatial and angular resolution is naturally built in standard plenoptic imaging. If N tot is the total number of pixels per line on the sensor, N x the number of microlenses per line, and N u the number of pixels per line associated with each microlens, then N x N u = N tot . Essentially, standard PI gives the same resolution and DOF one would obtain with a N u times smaller NA. The final advantage is thus practical rather than fundamental, and is limited * francesco.pepe@ba.infn.it † milena.dangelo@uniba.it to higher luminosity (hence SNR) of the final image and parallel acquisition of m...

show abstract

mentioning

confidence: 99%

Diffraction-Limited Plenoptic Imaging with Correlated Light

et al. 2017

View full text Add to dashboard Cite

show abstract

“…The FOV that we metrically calibrated previously is 50×50×30mm [23]. This smaller FOV is chosen in the center of the image plane for increased accuracy.…”

Section: Methodsmentioning

confidence: 99%

Biocompatible Near-Infrared Three-Dimensional Tracking System

Decker

Shademan

Opfermann

et al. 2017

IEEE Trans. Biomed. Eng.

Self Cite

View full text Add to dashboard Cite

A fundamental challenge in soft-tissue surgery is that target tissue moves and deforms, becomes occluded by blood or other tissue, and is difficult to differentiate from surrounding tissue. We developed small biocompatible near-infrared fluorescent (NIRF) markers with a novel fused plenoptic and NIR camera tracking system, enabling 3D tracking of tools and target tissue while overcoming blood and tissue occlusion in the uncontrolled, rapidly changing surgical environment. In this work, we present the tracking system and marker design and compare tracking accuracies to standard optical tracking methods using robotic experiments. At speeds of 1 mm/s, we observe tracking accuracies of 1.61 mm, degrading only to 1.71 mm when the markers are covered in blood and tissue.

show abstract

“…Despite its limitation, PI is currently employed in the most diverse applications, including 3D imaging and sensing [5,22], stereoscopy [2,23,24], particle image velocimetry [25], particle tracking and sizing [26], wavefront sensing [6,[27][28][29], microscopy [4,6,11,30] and digital cameras with refocusing capabilities [31]. Plenoptic imaging has also been employed in surgical robotics [32], endoscopic applications [33], and blood-flow visualization [34].…”

Section: Introductionmentioning

confidence: 99%

Correlation Plenoptic Imaging: An Overview

et al. 2018

View full text Add to dashboard Cite

Plenoptic imaging (PI) enables refocusing, depth-of-field (DOF) extension and 3D visualization, thanks to its ability to reconstruct the path of light rays from the lens to the image. However, in state-of-the-art plenoptic devices, these advantages come at the expenses of the image resolution, which is always well above the diffraction limit defined by the lens numerical aperture (NA). To overcome this limitation, we have proposed exploiting the spatio-temporal correlations of light, and to modify the ghost imaging scheme by endowing it with plenoptic properties. This approach, named Correlation Plenoptic Imaging (CPI), enables pushing both resolution and DOF to the fundamental limit imposed by wave-optics. In this paper, we review the methods to perform CPI both with chaotic light and with entangled photon pairs. Both simulations and a proof-of-principle experimental demonstration of CPI will be presented.

show abstract

Plenoptic cameras in surgical robotics: Calibration, registration, and evaluation

Cited by 22 publications

References 26 publications

Diffraction-Limited Plenoptic Imaging with Correlated Light

Diffraction-Limited Plenoptic Imaging with Correlated Light

Biocompatible Near-Infrared Three-Dimensional Tracking System

Correlation Plenoptic Imaging: An Overview

Contact Info

Product

Resources

About