Exploring Simple and Transferable Recognition-Aware Image Processing

Liu, Zhuang; Wang, Hung-Ju; Zhou, Tao; Shen, Zhi-Qiang; Kang, Bingyi; Shelhamer, Evan; Darrell, Trevor

doi:10.48550/arxiv.1910.09185

Cited by 5 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(b) Dirty Pixels [26], same as Vanilla denoiser + classifier, but trained end-to-end using the QIS noisy data. (c) Restoration Network [30], [31], which trains a denoiser but uses a pretrained classifier. This can be viewed as a middle-ground solution between Vanilla and Dirty Pixels.…”

Section: Competing Methods and Our Networkmentioning

confidence: 99%

“…They observed that less aggressive denoisers are better for classification because the features are preserved. Other methods adopt similar strategies, e.g., using discrete cosine transform [27], training a classifier to help denoising [28] or using an ensemble method [29], or training a denoiser that are better suited for pre-trained classifiers [30], [31].…”

Section: Prior Workmentioning

confidence: 99%

See 1 more Smart Citation

Image Classification in the Dark using Quanta Image Sensors

Gnanasambandam¹,

Chan²

2020

Preprint

View full text Add to dashboard Cite

State-of-the-art image classifiers are trained and tested using well-illuminated images. These images are typically captured by CMOS image sensors with at least tens of photons per pixel. However, in dark environments when the photon flux is low, image classification becomes difficult because the measured signal is suppressed by noise. In this paper, we present a new low-light image classification solution using Quanta Image Sensors (QIS). QIS are a new type of image sensors that possess photon counting ability without compromising on pixel size and spatial resolution. Numerous studies over the past decade have demonstrated the feasibility of QIS for low-light imaging, but their usage for image classification has not been studied. This paper fills the gap by presenting a student-teacher learning scheme which allows us to classify the noisy QIS raw data. We show that with student-teacher learning, we are able to achieve image classification at a photon level of one photon per pixel or lower. Experimental results verify the effectiveness of the proposed method compared to existing solutions.

show abstract

Section: Competing Methods and Our Networkmentioning

confidence: 99%

Section: Prior Workmentioning

confidence: 99%

Image Classification in the Dark using Quanta Image Sensors

Gnanasambandam¹,

Chan²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent research on recognition-aware image processing has shown promising results on improving accuracy of classification models and simultaneously preserving the perceptual quality [19,28]. This class of methods keep the classification model fixed, and only train the enhancement module.…”

Section: Plcc ↑ Bicubic Resizermentioning

confidence: 99%

“…Recently Liu et al [19] proposed an approach to improve machine interpretability of images by optimizing the recognition loss on the image processing network. They study super-resolution, denoising, and JPEG-deblocking as preprocessing operations, and show that the recognition performance gain can transfer when evaluated on different architectures and tasks.…”

Section: Related Workmentioning

confidence: 99%

“…They study super-resolution, denoising, and JPEG-deblocking as preprocessing operations, and show that the recognition performance gain can transfer when evaluated on different architectures and tasks. Our approach differs with [19] in that (1) we exclusively focus on improving the recognition performance, regardless of the perceptual quality, and (2) our pre-processing resizer is jointly trained with the recognition model, and consequently better adapts to the recognition architecture. This means the proposed model is not constrained to learn specific enhancements (e.g.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Resize Images for Computer Vision Tasks

Talebi¹,

Milanfar²

2021

Preprint

View full text Add to dashboard Cite

For all the ways convolutional neural nets have revolutionized computer vision in recent years, one important aspect has received surprisingly little attention: the effect of image size on the accuracy of tasks being trained for. Typically, to be efficient, the input images are resized to a relatively small spatial resolution (e.g. 224 × 224), and both training and inference are carried out at this resolution. The actual mechanism for this re-scaling has been an afterthought: Namely, off-the-shelf image resizers such as bilinear and bicubic are commonly used in most machine learning software frameworks. But do these resizers limit the on-task performance of the trained networks? The answer is yes. Indeed, we show that the typical linear resizer can be replaced with learned resizers that can substantially improve performance. Importantly, while the classical resizers typically result in better perceptual quality of the downscaled images, our proposed learned resizers do not necessarily give better visual quality, but instead improve task performance.Our learned image resizer is jointly trained with a baseline vision model. This learned CNN-based resizer creates machine friendly visual manipulations that lead to a consistent improvement of the end task metric over the baseline model. Specifically, here we focus on the classification task with the ImageNet dataset [25], and experiment with four different models to learn resizers adapted to each model. Moreover, we show that the proposed resizer can also be useful for fine-tuning the classification baselines for other vision tasks. To this end, we experiment with three different baselines to develop image quality assessment (IQA) models on the AVA dataset [23].

show abstract

MaxViT: Multi-axis Vision Transformer

Talebi

Zhang

et al. 2022

Lecture Notes in Computer Science

325

View full text Add to dashboard Cite

Image resizing operation is a fundamental preprocessing module in modern computer vision. Throughout the deep learning revolution, researchers have overlooked the potential of alternative resizing methods beyond the commonly used resizers that are readily available, such as nearestneighbors, bilinear, and bicubic. The key question of our interest is whether the front-end resizer affects the performance of deep vision models? In this paper, we present an extremely lightweight multilayer Laplacian resizer with only a handful of trainable parameters, dubbed MULLER resizer. MULLER has a bandpass nature in that it learns to boost details in certain frequency subbands that benefit the downstream recognition models. We show that MULLER can be easily plugged into various training pipelines, and it effectively boosts the performance of the underlying vision task with little to no extra cost. Specifically, we select a state-of-the-art vision Transformer, MaxViT [50], as the baseline, and show that, if trained with MULLER, MaxViT gains up to 0.6% top-1 accuracy, and meanwhile enjoys 36% inference cost saving to achieve similar top-1 accuracy on ImageNet-1k, as compared to the standard training scheme. Notably, MULLER's performance also scales with model size and training data size such as ImageNet-21k and JFT, and it is widely applicable to multiple vision tasks, including image classification, object detection and segmentation, as well as image quality assessment.

show abstract

Exploring Simple and Transferable Recognition-Aware Image Processing

Cited by 5 publications

References 29 publications

Image Classification in the Dark using Quanta Image Sensors

Image Classification in the Dark using Quanta Image Sensors

Learning to Resize Images for Computer Vision Tasks

MaxViT: Multi-axis Vision Transformer

Contact Info

Product

Resources

About