In this paper, we present the architecture of a smart imaging sensor (SIS) for face recognition, based on a custom-design smart pixel capable of computing local spatial gradients in the analog domain, and a digital coprocessor that performs image classification. The SIS uses spatial gradients to compute a lightweight version of local binary patterns (LBP), which we term ringed LBP (RLBP). Our face recognition method, which is based on Ahonen’s algorithm, operates in three stages: (1) it extracts local image features using RLBP, (2) it computes a feature vector using RLBP histograms, (3) it projects the vector onto a subspace that maximizes class separation and classifies the image using a nearest neighbor criterion. We designed the smart pixel using the TSMC 0.35 μm mixed-signal CMOS process, and evaluated its performance using postlayout parasitic extraction. We also designed and implemented the digital coprocessor on a Xilinx XC7Z020 field-programmable gate array. The smart pixel achieves a fill factor of 34% on the 0.35 μm process and 76% on a 0.18 μm process with 32 μm × 32 μm pixels. The pixel array operates at up to 556 frames per second. The digital coprocessor achieves 96.5% classification accuracy on a database of infrared face images, can classify a 150×80-pixel image in 94 μs, and consumes 71 mW of power.

No abstract

Object location is a crucial computer vision method often used as a previous stage to object classification. Object-location algorithms require high computational and memory resources, which poses a difficult challenge for portable and low-power devices, even when the algorithm is implemented using dedicated digital hardware. Moving part of the computation to the imager may reduce the memory requirements of the digital post-processor and exploit the parallelism available in the algorithm. This paper presents the architecture of a Smart Imaging Sensor (SIS) that performs object location using pixel-level parallelism. The SIS is based on a custom smart pixel, capable of computing frame differences in the analog domain, and a digital coprocessor that performs morphological operations and connected components to determine the bounding boxes of the detected objects. The smart-pixel array implements on-pixel temporal difference computation using analog memories to detect motion between consecutive frames. Our SIS can operate in two modes: (1) as a conventional image sensor and (2) as a smart sensor which delivers a binary image that highlights the pixels in which movement is detected between consecutive frames and the object bounding boxes. In this paper, we present the design of the smart pixel and evaluate its performance using post-parasitic extraction on a 0.35 µm mixed-signal CMOS process. With a pixel-pitch of 32 µm × 32 µm, we achieved a fill factor of 28%. To evaluate the scalability of the design, we ported the layout to a 0.18 µm process, achieving a fill factor of 74%. On an array of 320×240 smart pixels, the circuit operates at a maximum frame rate of 3846 frames per second. The digital coprocessor was implemented and validated on a Xilinx Artix-7 XC7A35T field-programmable gate array that runs at 125 MHz, locates objects in a video frame in 0.614 µs, and has a power consumption of 58 mW.

No abstract

Images produced by CMOS sensors may contain defective pixels due to noise, manufacturing errors, or device malfunction, which must be detected and corrected at early processing stages in order to produce images that are useful to human users and image-processing or machine-vision algorithms. This paper proposes a defective pixel detection and correction algorithm and its implementation using CMOS analog circuits, which are integrated with the image sensor at the pixel and column levels. During photocurrent integration, the circuit detects defective values in parallel at each pixel using simple arithmetic operations within a neighborhood. At the image-column level, the circuit replaces the defective pixels with the median value of their neighborhood. To validate our approach, we designed a 128×128-pixel imager in a 0.35μm CMOS process, which integrates our defective-pixel detection/correction circuits and processes images at 694 frames per second, according to post-layout simulations. Operating at that frame rate, our proposed algorithm and its CMOS implementation produce better results than current state-of-the-art algorithms: it achieves a Peak Signal to Noise Ratio (PSNR) and Image Enhancement Factor (IEF) of 45 dB and 198.4, respectively, in images with 0.5% random defective pixels, and a PSNR of 44.4 dB and IEF of 194.2, respectively, in images with 1.0% random defective pixels.