Neuromorphic Camera Guided High Dynamic Range Imaging

Han, Jin; Zhou, Chu; Duan, Peiqi; Xu, Chang; Xu, Chao; Huang, Tiejun; Shi, Boxin

doi:10.1109/cvpr42600.2020.00180

Cited by 87 publications

(65 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One example of neuromorphic chip design indicates that the ADC circuitry block takes a chip area of 0.021 mm 2 using 90 nm technology while the advanced spintronics method using 32 nm technology could reduce the ADC to 10 lm 2 [36,37]. Image sensors with off-chip and on-chip options are both being considered [47,48]. Combining hardware design options and various operating algorithms, the optimization of performance, feature, size, and power consumption could ultimately determine chip-stacking architectures.…”

Section: Infraredmentioning

confidence: 99%

Advancement of Chip Stacking Architectures and Interconnect Technologies for Image Sensors

Lu¹

2021

Journal of Electronic Packaging

View full text Add to dashboard Cite

Numerous technology breakthroughs have been made in image sensor development in the past two decades. Image sensors have evolved into a technology platform to support many applications. Their successful implementation in mobile devices has accelerated market demand and established a business platform to propel continuous innovation and performance improvement extending to surveillance, medical, and automotive industries. This overview briefs the general camera module and the crucial technology elements of chip stacking architectures and advanced interconnect technologies. This study will also examine the role of pixel electronics in determining the chip stacking architecture and interconnect technology of choice. It is conducted by examining a few examples of CMOS Image Sensors (CIS) for different functions such as visible light detection, Single Photon Avalanche Photodiode (SPAD) for low light detection, rolling shutter and global shutter, and depth sensing and Light Detection And Ranging (LiDAR). Performance attributes of different architectures of chip stacking are overviewed. Direct bonding followed by Via-last through silicon via (Via-last TSV) and hybrid bonding (HB) technologies are identified as newer and favorable chip-to-chip interconnect technologies for image sensor chip stacking. The state-of-the-art ultra-high-density interconnect manufacturability is also highlighted.

show abstract

Section: Infraredmentioning

confidence: 99%

Advancement of Chip Stacking Architectures and Interconnect Technologies for Image Sensors

Lu¹

2021

Journal of Electronic Packaging

View full text Add to dashboard Cite

show abstract

“…Let x k,v and y k,v be the pixels at position (k, v) in the input and output of SVC, respectively. w (1) , w (2) , w (3) , w (4) , w (5) , w (6) , w (7) , and w (8) represent different convolution kernels. The proposed SVC is then defined as follows:…”

Section: Fig 6: Illustration Of Four Different Patterns In a 5×5 Rece...mentioning

confidence: 99%

Deep Joint Demosaicing and High Dynamic Range Imaging within a Single Shot

Xu,

Liu,

et al. 2021

Preprint

View full text Add to dashboard Cite

Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVEbased HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over-and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying exposures. For the latter one, an exposure-guidance method is proposed against the interference from over-and under-exposed pixels. Finally, a joint demosaicing and HDRI deep learning framework is formalized to include the two novel components and to realize an end-toend single-shot HDRI. Experiments indicate that the proposed end-to-end framework avoids the problem of cumulative errors and surpasses the related state-of-the-art methods.

show abstract

“…In contrast to image/video reconstruction from events, [19] proposed to generate events from video frames. Some other works also explored the potential of events for image deblurring [26,32,67], HDR imaging [25,81], and event denoising [3]. For more details about event-based vision, refer to a survey [16].…”

Section: Related Workmentioning

confidence: 99%

EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation

Wang¹,

Chae²,

Yoon³

et al. 2021

Preprint

View full text Add to dashboard Cite

Event cameras sense per-pixel intensity changes and produce asynchronous event streams with high dynamic range and less motion blur, showing advantages over the conventional cameras. A hurdle of training event-based models is the lack of large qualitative labeled data. Prior works learning end-tasks mostly rely on labeled or pseudolabeled datasets obtained from the active pixel sensor (APS) frames; however, such datasets' quality is far from rivaling those based on the canonical images. In this paper, we propose a novel approach, called EvDistill, to learn a student network on the unlabeled and unpaired event data (target modality) via knowledge distillation (KD) from a teacher network trained with large-scale, labeled image data (source modality). To enable KD across the unpaired modalities, we first propose a bidirectional modality reconstruction (BMR) module to bridge both modalities and simultaneously exploit them to distill knowledge via the crafted pairs, causing no extra computation in the inference. The BMR is improved by the end-tasks and KD losses in an end-to-end manner. Second, we leverage the structural similarities of both modalities and adapt the knowledge by matching their distributions. Moreover, as most prior feature KD methods are uni-modality and less applicable to our problem, we propose to leverage an affinity graph KD loss to boost the distillation. Our extensive experiments on semantic segmentation and object recognition demonstrate that EvDistill achieves significantly better results than the prior works and KD with only events and APS frames.

show abstract

Neuromorphic Camera Guided High Dynamic Range Imaging

Cited by 87 publications

References 33 publications

Advancement of Chip Stacking Architectures and Interconnect Technologies for Image Sensors

Advancement of Chip Stacking Architectures and Interconnect Technologies for Image Sensors

Deep Joint Demosaicing and High Dynamic Range Imaging within a Single Shot

EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation

Contact Info

Product

Resources

About