Estimates of the information content and dimensionality of natural scenes from proximity distributions

Abstract

Natural scenes, like most all natural data sets, show considerable redundancy. Although many forms of redundancy have been investigated (e.g., pixel distributions, power spectra, contour relationships, etc.), estimates of the true entropy of natural scenes have been largely considered intractable. We describe a technique for estimating the entropy and relative dimensionality of image patches based on a function we call the proximity distribution (a nearest-neighbor technique). The advantage of this function over simple statistics such as the power spectrum is that the proximity distribution is dependent on all forms of redundancy. We demonstrate that this function can be used to estimate the entropy (redundancy) of 3 ϫ 3 patches of known entropy as well as 8 ϫ 8 patches of Gaussian white noise, natural scenes, and noise with the same power spectrum as natural scenes. The techniques are based on assumptions regarding the intrinsic dimensionality of the data, and although the estimates depend on an extrapolation model for images larger than 3 ϫ 3, we argue that this approach provides the best current estimates of the entropy and compressibility of natural-scene patches and that it provides insights into the efficiency of any coding strategy that aims to reduce redundancy. We show that the sample of 8 ϫ 8 patches of natural scenes used in this study has less than half the entropy of 8 ϫ 8 white noise and less than 60% of the entropy of noise with the same power spectrum. In addition, given a finite number of samples ͑Ͻ2 20 ͒ drawn randomly from the space of 8 ϫ 8 patches, the subspace of 8 ϫ 8 natural-scene patches shows a dimensionality that depends on the sampling density and that for low densities is significantly lower dimensional than the space of 8 ϫ 8 patches of white noise and noise with the same power spectrum.

80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition

“…This is a very large space -if each dimension has 8 bits, there are a total of 10 7400 possible images. This is a huge number, especially if we consider that a human in a 100 years only gets to see 10 11 frames (at 30 frames/second). However, natural images only correspond to a tiny fraction of this space (most of the images correspond to white noise), and it is natural to investigate the size of that fraction.…”

“…However, natural images only correspond to a tiny fraction of this space (most of the images correspond to white noise), and it is natural to investigate the size of that fraction. A number of studies [10], [25] have been devoted to characterize the space of natural images by studying the statistics of small image patches. However, low-resolution scenes are quite different to patches extracted by randomly cropping small patches from images.…”

“…Note that we are concerned solely with recognition performance, not with issues of intrinsic dimensionality or the like as explored in other studies of large collection of image patches [10], [25]. In this section, we explore how the probability of finding images with a similar label nearby increases with the size of the dataset.…”

Patch Complexity, Finite Pixel Correlations and Optimal Denoising

“…Characterizing the properties of natural images is critical for computer and human vision [18,13,20,16,7,23]. In particular, low level vision tasks such as denoising, super resolution, deblurring and completion, are fundamentally ill-posed since an infinite number of images x can explain an observed degraded image y.…”

“…However, while most image priors (parametric, non-parametric, learning-based) [2,14,20,16,23] as well as studies on image statistics [13,7] are restricted to local image patches or kernels, little is known about their dependence on patch size. Hence another question of practical importance is the following: What is the potential restoration gain from an increase in patch size?…”