VisionISP: Repurposing the Image Signal Processor for Computer Vision Applications

Wu, Chyuan-Tyng; Isikdogan, Leo F.; Sushma, Rao; Nayak, Bhavin V.; Gerasimow, Timo; Sutic, Aleksandar; Ain-kedem, Liron; Michael, Gilad

doi:10.1109/icip.2019.8803607

Cited by 26 publications

(12 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ISP has the information that can explain the image variation and computer vision can learn to compensate through that variation. Through this, computer vision can complement the function of ISP and if the function of ISP is used for low-level operations such as denosing, and computer vision is used for high-level operation; this can secure capacity and lower processing power [ 17 ].…”

Section: Methodsmentioning

confidence: 99%

Image Pre-Processing Method of Machine Learning for Edge Detection with Image Signal Processor Enhancement

2021

View full text Add to dashboard Cite

Even though computer vision has been developing, edge detection is still one of the challenges in that field. It comes from the limitations of the complementary metal oxide semiconductor (CMOS) Image sensor used to collect the image data, and then image signal processor (ISP) is additionally required to understand the information received from each pixel and performs certain processing operations for edge detection. Even with/without ISP, as an output of hardware (camera, ISP), the original image is too raw to proceed edge detection image, because it can include extreme brightness and contrast, which is the key factor of image for edge detection. To reduce the onerousness, we propose a pre-processing method to obtain optimized brightness and contrast for improved edge detection. In the pre-processing, we extract meaningful features from image information and perform machine learning such as k-nearest neighbor (KNN), multilayer perceptron (MLP) and support vector machine (SVM) to obtain enhanced model by adjusting brightness and contrast. The comparison results of F1 score on edgy detection image of non-treated, pre-processed and pre-processed with machine learned are shown. The pre-processed with machine learned F1 result shows an average of 0.822, which is 2.7 times better results than the non-treated one. Eventually, the proposed pre-processing and machine learning method is proved as the essential method of pre-processing image from ISP in order to gain better edge detection image. In addition, if we go through the pre-processing method that we proposed, it is possible to more clearly and easily determine the object required when performing auto white balance (AWB) or auto exposure (AE) in the ISP. It helps to perform faster and more efficiently through the proactive ISP.

show abstract

Section: Methodsmentioning

confidence: 99%

Image Pre-Processing Method of Machine Learning for Edge Detection with Image Signal Processor Enhancement

2021

View full text Add to dashboard Cite

show abstract

“…They train with simulated RAW images. Wu et al [17] suggested VisionISP, a trainable ISP, which is trained to optimize object detection in an autonomous driving setting. In this work, instead of designing or training an image processing module, i.e., a module whose input and output are images, we are focused on the case that the vision model is operating on RAW images.…”

Section: Related Workmentioning

confidence: 99%

“…This happens because the ISP serves as a 'normalization' of the data, transforming it to a canonical space, which is independent (or less dependent) of the camera used or capturing environment. Solutions discussed in the literature usually involve an adjustment of the ISP for the vision task, either manually [18] or via learning [17]. Our method deals with the case where the ISP is discarded and mitigates the drop in performance.…”

Section: Introductionmentioning

confidence: 99%

ISP Distillation

Schwartz¹,

Bronstein²,

Giryes³

2021

Preprint

View full text Add to dashboard Cite

Nowadays, many of the images captured are "observed" by machines only and not by humans, for example, robots' or autonomous cars' cameras. High-level machine vision models, such as object recognition, assume images are transformed to some canonical image space by the camera ISP. However, the camera ISP is optimized for producing visually pleasing images to human observers and not for machines, thus, one may spare the ISP compute time and apply the vision models directly on the raw data. Yet, it has been shown that training such models directly on the RAW images results in a performance drop. To mitigate this drop in performance (without the need to annotate RAW data), we use a dataset of RAW and RGB image pairs, which can be easily acquired with no human labeling. We then train a model that is applied directly on the RAW data by using knowledge distillation such that the model predictions for RAW images will be aligned with the predictions of an off-the-shelf pre-trained model for processed RGB images. Our experiments show that our performance on RAW images is significantly better than a model trained on labeled RAW images. It also reasonably matches the predictions of a pre-trained model on processed RGB images, while saving the ISP compute overhead.

show abstract

“…Recently even standard image signal processing (ISP) chips include neural co-processors and, of course, many companies are developing increasingly advanced chips. At the time of this writing neural image analysis is increasingly a standard component camera component, especially in mobile devices [83,84] and proposals have emerged for ISP chips designed specifically for neural processing [85,86].…”

Section: Cameras and Artificial Neural Networkmentioning

confidence: 99%

Smart Cameras

Brady,

Hu,

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

We review camera architecture in the age of artificial intelligence. Modern cameras use physical components and software to capture, compress and display image data. Over the past 5 years, deep learning solutions have become superior to traditional algorithms for each of these functions. Deep learning enables 10-100x reduction in electrical sensor power per pixel, 10x improvement in depth of field and dynamic range and 10-100x improvement in image pixel count. Deep learning enables multiframe and multiaperture solutions that fundamentally shift the goals of physical camera design. Here we review the state of the art of deep learning in camera operations and consider the impact of AI on the physical design of cameras.

show abstract

VisionISP: Repurposing the Image Signal Processor for Computer Vision Applications

Cited by 26 publications

References 17 publications

Image Pre-Processing Method of Machine Learning for Edge Detection with Image Signal Processor Enhancement

Image Pre-Processing Method of Machine Learning for Edge Detection with Image Signal Processor Enhancement

ISP Distillation

Smart Cameras

Contact Info

Product

Resources

About