Image sensing with multilayer nonlinear optical neural networks

Wang, Tianyu; Sohoni, Mandar M.; Wright, Logan G.; Stein, Martin M.; Ma, Shi-Yuan; Onodera, Tatsuhiro; Anderson, Maxwell G.; McMahon, Peter L.

doi:10.1038/s41566-023-01170-8

Cited by 90 publications

(15 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The high-speed SMUX-STPC network will revolutionize video-based real-world applications, for example, biological research ( 52 ). In biology, tracking cell movements is a fundamental procedure that helps with analyzing tissue development and diseases ( 53 ).…”

Section: Resultsmentioning

confidence: 99%

Ultrafast dynamic machine vision with spatiotemporal photonic computing

Zhou

Zhang

et al. 2023

Sci. Adv.

View full text Add to dashboard Cite

Ultrafast dynamic machine vision in the optical domain can provide unprecedented perspectives for high-performance computing. However, owing to the limited degrees of freedom, existing photonic computing approaches rely on the memory’s slow read/write operations to implement dynamic processing. Here, we propose a spatiotemporal photonic computing architecture to match the highly parallel spatial computing with high-speed temporal computing and achieve a three-dimensional spatiotemporal plane. A unified training framework is devised to optimize the physical system and the network model. The photonic processing speed of the benchmark video dataset is increased by 40-fold on a space-multiplexed system with 35-fold fewer parameters. A wavelength-multiplexed system realizes all-optical nonlinear computing of dynamic light field with a frame time of 3.57 nanoseconds. The proposed architecture paves the way for ultrafast advanced machine vision free from the limits of memory wall and will find applications in unmanned systems, autonomous driving, ultrafast science, etc.

show abstract

Section: Resultsmentioning

confidence: 99%

Ultrafast dynamic machine vision with spatiotemporal photonic computing

Zhou

Zhang

et al. 2023

Sci. Adv.

View full text Add to dashboard Cite

show abstract

“…In this work, we explore the potential for analog photonics to improve throughput and power consumption in an image acquisition pipeline after image formation on the focal plane array. This is in stark contrast to the significant body of work focused on image classification, inference, or compressed sensing of the raw scene information that operates by processing the scene data directly in the analog optical domain at the image acquisition stage 4 , 7 – 11 . While both schemes are valuable, we believe there are several important advantages for our approach, which has been relatively under-explored.…”

Section: Introductionmentioning

confidence: 98%

“…With the development of solid-state focal planes and digital coding, however, this function changed, and modern digital cameras act as transceivers that transform massively parallel optical data streams into serial coded electronic data that is processed 1 pixel at a time 1 – 3 . While this paradigm shift introduced numerous advantages, the power consumption associated with electronic digital processing, along with limits on data transmission rate and storage capacity, are now the major bottlenecks limiting image data acquisition rates 1 – 4 . In contrast, compact lens designs are capable of resolving greater than 10 gigapixels of transverse resolution 5 , 6 , while advances in multimodal imaging systems capable of acquiring spectral, polarization, temporal, and range information could enable future imaging systems to acquire (Tera)10 12 pixels per second of data.…”

Section: Introductionmentioning

confidence: 99%

Integrated photonic encoder for low power and high-speed image processing

Wang,

Redding,

Karl

et al. 2024

Nat Commun

View full text Add to dashboard Cite

Modern lens designs are capable of resolving greater than 10 gigapixels, while advances in camera frame-rate and hyperspectral imaging have made data acquisition rates of Terapixel/second a real possibility. The main bottlenecks preventing such high data-rate systems are power consumption and data storage. In this work, we show that analog photonic encoders could address this challenge, enabling high-speed image compression using orders-of-magnitude lower power than digital electronics. Our approach relies on a silicon-photonics front-end to compress raw image data, foregoing energy-intensive image conditioning and reducing data storage requirements. The compression scheme uses a passive disordered photonic structure to perform kernel-type random projections of the raw image data with minimal power consumption and low latency. A back-end neural network can then reconstruct the original images with structural similarity exceeding 90%. This scheme has the potential to process data streams exceeding Terapixel/second using less than 100 fJ/pixel, providing a path to ultra-high-resolution data and image acquisition systems.

show abstract

“…Also, extracting more information from less photons alleviates the demands on optical-todigital signal processing speed and data storage volume. Thus far, compressive sensing systems have employed dynamic optical modulations using Spatial Light Modulators [1][2][3][4] , Digital Micromirror Devices 5,6 , coded apertures [7][8][9] , variational quantum sensors 10 , diffractive optical networks [11][12][13][14][15] , optical encoders 16 , and so on. Despite encouraging advancement, these modules are limited in speed, which restricts their use in fast-moving environments where the whole-scene information must be captured in a fraction of a second.…”

Section: Introductionmentioning

confidence: 99%

Physics-Informed Masked Autoencoder for Active Sparse Imaging

McEvoy,

Tafone,

Sua

et al. 2024

Preprint

View full text Add to dashboard Cite

Imaging technology based on detecting individual photons has seen tremendous progress in recent years, with broad applications in autonomous driving, biomedical imaging, astronomical observation, and more. Comparing with conventional methods, however, it takes much longer time and relies on sparse and noisy photon-counting data to form an image. Here we introduce Physics-Informed Masked Autoencoder (PI-MAE) as a fast and efficient approach for data acquisition and image reconstruction through hardware implementation of the MAE (Masked Autoencoder). We examine its performance on a single-photon LiDAR system when trained on digitally masked MNIST data. Our results show that, with 1.8 × 10−6 or less detected photons per pulse and down to 9 detected photons per pixel, it achieves high-quality image reconstruction on unseen object classes with 90% physical masking. Our results highlight PI-MAE as a viable hardware accelerator for significantly improving the performance of single-photon imaging systems in photon-starving applications.

show abstract

Image sensing with multilayer nonlinear optical neural networks

Cited by 90 publications

References 49 publications

Ultrafast dynamic machine vision with spatiotemporal photonic computing

Ultrafast dynamic machine vision with spatiotemporal photonic computing

Integrated photonic encoder for low power and high-speed image processing

Physics-Informed Masked Autoencoder for Active Sparse Imaging

Contact Info

Product

Resources

About