Oil droplets are spherical organelles found in the cone photoreceptors of vertebrates. They are generally assumed to focus incident light into the outer segment, and thereby improve light catch because of the droplets' spherical lens-like shape. However, using full-wave optical simulations of physiologically realistic cone photoreceptors from birds, frogs and turtles, we find that pigmented oil droplets actually drastically reduce the transmission of light into the outer segment integrated across the full visible wavelength range of each species. Only transparent oil droplets improve light catch into the outer segments, and any enhancement is critically dependent on the refractive index, diameter of the oil droplet, and diameter and length of the outer segment. Furthermore, oil droplets are not the only optical elements found in cone inner segments. The ellipsoid, a dense aggregation of mitochondria situated immediately prior to the oil droplet, mitigates the loss of light at the oil droplet surface. We describe a framework for integrating these optical phenomena into simple models of receptor sensitivity, and the relevance of these observations to evolutionary appearance and loss of oil droplets is discussed.