In-sensor optoelectronic computing using electrostatically doped silicon

Abstract

Complementary metal-oxide-semiconductor (CMOS) image sensors are a visual outpost of many machines that interact with the world. While they presently separate image capture in front-end silicon photodiode arrays from image processing in digital back-ends, efforts to process images within the photodiode array itself are rapidly emerging, in hopes of minimizing the data transfer between sensing and computing, and the associated overhead in energy and bandwidth. Electrical modulation, or programming, of photocurrents is requisite for such in-sensor computing, which was indeed demonstrated with electrostatically doped, but non-silicon, photodiodes. CMOS image sensors are currently incapable of in-sensor computing, as their chemically doped photodiodes cannot produce electrically tunable photocurrents. Here we report in-sensor computing with an array of electrostatically doped silicon p-i-n photodiodes, which is amenable to seamless integration with the rest of the CMOS image sensor electronics. This silicon-based approach could more rapidly bring in-sensor computing to the real world due to its compatibility with the mainstream CMOS electronics industry. Our wafer-scale production of thousands of silicon photodiodes using standard fabrication emphasizes this compatibility. We then demonstrate in-sensor processing of optical images using a variety of convolutional lters electrically programmed into a 3 × 3 network of these photodiodes. Main TextComplementary metal oxide semiconductor (CMOS) image sensors have become an indispensable part of our data-driven world, where visual information prevails 1,2 . The front-end silicon photodiode array in a CMOS image sensor converts light into electrical currents. These electrical data undergo analog-to-digital conversion and are then shuttled to a digital back-end for image processing. While this standard sequence of front-end image capture and back-end processing restricts the role of the photodiode array to sensing, emerging machine vision applications would bene t from data processing within the photodiode array itself 3,4 . For example, in object tracking for self-driving vehicles, drones, or robots, where only the edges of objects are relevant [5][6][7][8] , edge extraction in the front-end photodiode array would be much more economical in energy expenditure, processing latency, required bandwidth, and memory usage, as compared to transferring the whole image data containing super uous information to the back-end digital processor-only to extract the edges 9 . Such in-sensor computing would require an electrical modulation, or programming, of photocurrents. In fact, in-sensor computing has been recently demonstrated with electrostatically doped photodiodes whose photocurrents can be modulated with gate biasing 10,11 . These pioneering works have realized electrostatically doped photodiodes by gating two-dimensional (2D) transition metal dichalcogenide (TMD) layers or their van der Waals (vdW) stacks [12][13][14] . In contrast, such in-sensor computing is not possible with ...

Nanomaterial-Based Synaptic Optoelectronic Devices for In-Sensor Preprocessing of Image Data

“…In contrast, synaptic optoelectronic devices perform image preprocessing at the image-sensor level . Through in-sensor preprocessing performed by the image-sensing device itself, the preprocessed data can be obtained from massive raw image data without computationally expensive preprocessing steps . In the following sections, we will describe advantages of the in-sensor preprocessing in more detail.…”

“…The roles of synaptic optoelectronic devices have been expanding to image filtering ,, and pattern classification, , which were originally achieved using conventional microprocessors with appropriate software. Synaptic optoelectronic devices can serve as image filters and ANNs, which can be used for feature extraction and image recognition. , For example, Jang et al reported a 32 × 32 MoS 2 phototransistor array (Figure a), which captures incoming images as image sensors and also functions as a matrix multiplication engine for image filtering and recognition .…”

“…Classification by Synaptic Optoelectronic Devices. The roles of synaptic optoelectronic devices have been expanding to image filtering 5,14,127 and pattern classification, 8,16 which were originally achieved using conventional microprocessors with appropriate software. Synaptic optoelectronic devices can serve as image filters and ANNs, which can be used for feature extraction and image recognition.…”

“…Recent advances in image acquisition − and data processing , technologies have enabled diverse machine vision applications including facile object detection − and accurate image recognition, , leading to a new era in the development of surveillance, self-driving vehicles, and autonomous robotics technologies. , Simultaneously, the amount of image data to be acquired and processed by the image sensing and processing device for realizing the machine vision application has been exponentially growing, significantly increasing the computational burden. , However, the conventional architecture of the machine vision system, where the front-end image sensor and the back-end processor are physically separated, involves inefficiency in terms of power consumption and data latency . This is because the considerable amount of image data in the entire time domain should be acquired by the image sensor and then transferred to the processor for image data processing and recognition, consequently consuming significant energy and processing time. , …”

“…One promising solution for this inefficiency that originated from the system configuration is to adopt a novel image-sensing and data-processing platform that emulates the human vision and image recognition system. ,, The human retina performs first-stage image preprocessing by the bipolar and ganglion cells as well as performs image acquisition by the photoreceptor cells. , Inspired by this image acquisition and preprocessing mechanism of the retina, researchers have made significant efforts to add the image data preprocessing functions (e.g., contrast enhancement , and image filtering , ) into the image sensors, intending to integrate the front-end image preprocessing function into the image sensing device . Such image data preprocessing in the image sensor is called in-sensor preprocessing, , where the image data can be preprocessed simultaneously during the image-sensing step without additional computations, thus reducing the computational burden in the image recognition step. , …”