Multi-aperture foveated imaging

Abstract

Foveated imaging, such as that evolved by biological systems to provide high angular resolution with a reduced space-bandwidth product, also offers advantages for manmade task-specific imaging. Foveated imaging systems using exclusively optical distortion are complex, bulky, and high cost, however. We demonstrate foveated imaging using a planar array of identical cameras combined with a prism array and superresolution reconstruction of a mosaicked image with a foveal variation in angular resolution of 5.9:1 and a quadrupling of the field of view. The combination of low-cost, mass-produced cameras and optics with computational image recovery offers enhanced capability of achieving large foveal ratios from compact, low-cost imaging systems. Conventional approaches to imaging typically aim for an approximately uniform spatial sampling frequency across the field of view, but for many applications, such as targeting, the salient requirement is for high-resolution imaging within a central, so-called foveal region of the image combined with a lowresolution periphery providing situational awareness and context. Foveated imaging offers more efficient use of a limited number of detector pixels or can be implemented as an image processing technique applied to conventional images to improve the efficiency of information transmission. Here our emphasis is to attain a large ratio between the spatial sampling frequency and image acuity for the central field of view (FOV) and a reduced sampling frequency at larger field angles. This mimics biological systems, such as the human visual system, where foveated imaging is associated with a variation in photoreceptor packing density that approximately mirrors the angular variation in the optical resolution of the eye.Imaging systems with a variation in magnification of up to a factor of 2 between a central FOV and the periphery (the foveal ratio) have been demonstrated using conventional optical approaches, but higher ratios require dramatic increases in optical complexity. Higher foveal ratios are attractive for a wide range of applications [1][2][3][4][5][6], and previous approaches include the use of multiresolution systems using single [1] or multiple sensors [2][3][4] and applications in microscopy [5,6]. In this Letter we report an experimental demonstration of computational construction of a foveated image using a multicamera array. Two mechanisms contribute to the high foveal ratio of 5.9:1 between the angular sampling frequency in the foveal and peripheral regions of the image: image distortion introduced by an array of prisms located in front of the camera array introduces nonuniform angular sampling by the sensor, and overlap of the field of regard of the cameras at the central FOV enables digital superresolution to increase the angular sampling rate at foveal regions. The use of mass-produced cameras and simple prisms enables high-performance foveated imaging at minimal cost.A 5 × 5 multicamera array is assembled on a single printed circuit board, with the relatively low ...

Superimposed multi-resolution imaging

“…Previous strategies to achieve foveated sampling include: the use of non-uniform sensors (with variable photoreceptor density, mimicking the variable sampling rate of the retina) [1]; optical distortion for foveated lens design [2][3][4]; computational integration of independent imagers with dissimilar resolutions [5][6][7][8][9] and the use of a single sensor segmented into multiple channels with dissimilar magnifications [10]. The high cost of hardware and the added complexity of non-uniform sensors, or the optical complexity of foveal optics design, usually make these solutions unattractive.…”

“…A different but related approach is to form a composite image from a mosaic of several images with dissimilarly optical distortions (e.g. using prisms) [9]. Computational integration of independent imaging systems is a simple, robust and powerful approach that enables arbitrary foveal ratios.…”

Foveation Pipeline for 360° Video-Based Telemedicine

“…Second, two or more cameras can be used and post-processed to generate a unified foveated image [9,10]. Though custom imaging sensors have been proposed [9,11], this approach also exploits off-the-shelf cameras [10]. We follow the latter approach, in agreement with Carles et al [6]'s argument that the low-cost, high-performance computing has made post-processing an appealing solution compared to the complexity of custom sensor and optics design.…”

A Hybrid Bionic Image Sensor Achieving FOV Extension and Foveated Imaging

“…In order to obtain both large FOV and high resolution, the Aware-2 imaging system [ 14 ] utilizes a multi-scale lens and a micro-camera array achieving 120° and 38 μrad instantaneous FOV of a single pixel, but the system is bulky, and it is time-consuming due to the required calibration of 98 micro-cameras and the sub-image mosaic with iterative methods. In addition, large volume redundant data results in low efficiency for object detection or target tracking [ 15 ]. In recent years, the development of a flexible printed circuit board and liquid lens has motivated some remarkable CACE designs [ 1 , 16 , 17 , 18 ].…”

“…Recently, some researchers have been studying the combination of a compound eye and foveated vision to improve the performance of imaging systems. Guillem Carles et al [ 15 ] developed a multichannel imaging system which combines prism array and PACE to obtain FOV extension and foveated imaging. However, in order to achieve a high foveal ratio in the fovea, the FOV of each ommatidium overlaps in the fovea region, and the system only achieves two-fold FOV extension with a 5 × 5 camera array.…”