The human retina contains five photoreceptor types: rods; short (S)-, mid (M)-, and long (L)-wavelength–sensitive cones; and melanopsin-expressing ganglion cells. Recently, it has been shown that selective increments in M-cone activation are paradoxically perceived as brightness decrements, as opposed to L-cone increments. Here we show that similar effects are also observed in the pupillary light response, whereby M-cone or S-cone increments lead to pupil dilation whereas L-cone or melanopic illuminance increments resulted in pupil constriction. Additionally, intermittent photoreceptor activation increased pupil constriction over a 30-min interval. Modulation of L-cone or melanopic illuminance within the 0.25–4-Hz frequency range resulted in more sustained pupillary constriction than light of constant intensity. Opposite results were found for S-cone and M-cone modulations (2 Hz), mirroring the dichotomy observed in the transient responses. The transient and sustained pupillary light responses therefore suggest that S- and M-cones provide inhibitory input to the pupillary control system when selectively activated, whereas L-cones and melanopsin response fulfill an excitatory role. These findings provide insight into functional networks in the human retina and the effect of color-coding in nonvisual responses to light, and imply that nonvisual and visual brightness discrimination may share a common pathway that starts in the retina.