Experience and experiments on human color constancy (i.e., Arend & Reeves, 1986; Craven & Foster, 1992) tell us that we are capable of judging the illumination. However, when asked to make a match of the illuminant's color and brightness, human observers seem to be quite poor (Granzier, Brenner, & Smeets, 2009a). Here we investigate whether human observers use (rather than match) daylight for estimating ecologically important dimensions: time of year, time of day, and outdoor temperature. In the first three experiments we had our observers evaluate calibrated color images of an outdoor urban scene acquired throughout a year. Although some observers could estimate the month and the temperature, overall they were quite poor at judging the time of day. In particular, observers were not able to discriminate between morning and afternoon pictures even when they were allowed to compare multiple images captured on the same day (Experiment 3). However, observers could distinguish between midday and sunset and sunrise daylight. Classification analysis showed that, given a perfect knowledge of its variation, an ideal observer could have performed the task over chance only considering the average chromatic variation in the picture. Instead, our observers reported using shadows to detect the position of the sun in order to estimate the time of day. However, this information is highly unreliable without knowledge of the orientation of the scene. In Experiment 4 we used an LED chamber in order to present our observers with lights whose chromaticity and illuminance varied along the daylight locus, thus isolating the light cues from the sun position cue. We conclude that discriminating the slight variations in chromaticity and brightness, which potentially distinguish morning and afternoon illuminations, lies beyond the ability of human observers.