In contrast to the classical findings of lightness constancy, recent psychophysical studies show the strong dependency of the perceived reflectance of a surface on the structure of the natural illumination. The present study examined this inconstancy for systematic variations in the light field and an image-based explanation for it. Observers matched the specular and diffuse reflectance of a three-dimensional object in a complex scene under a fixed light field to that in the scene under different light fields with variable mean, contrast, and gamma. For the both specular and diffuse components, the matched reflectance was relatively constant against changes in the mean illuminance but varied extensively with changes in the contrast and gamma of the light field. We found that the matching data were well predicted by the similarity of the subband histograms of the images. The results support the notion that early spatial filtering can provide a unified account of both the constancy in the perceived surface reflectance against mean illuminance and the inconstancy for higher-order illumination statistics.