Subjective visual experience builds on sensory encoding of light reflected by different objects in our environment. Most retinal ganglion cells encode changes in light intensity, quantified as contrast, rather than the absolute intensity. Mathematically, contrast is often defined as a relative change in light intensity. Activity in the visual system and perceptual responses are usually explained with such definitions of contrast. Here, for the first time, we explicitly explored how contrast is actually represented in the visual system. Using mouse retina electrophysiology, we show that response strength of OFF retinal ganglion cells does not represent relative, but absolute changes in light intensity. ON RGC response strength is governed by a combination of absolute and relative change in light intensity. This is true for a wide range of ambient light levels, at least from scotopic to high mesopic regimes. Consequently, light decrements and increments are represented asymmetrically in the retina, which may explain the asymmetries in responses to negative and positive contrast observed throughout the visual system. These findings may help to more thoroughly design and interpret vision science studies where responses are driven by contrast of the visual stimuli.