Color and polarization provide complementary information about the world and are detected by specialized photoreceptors. However, the downstream neural circuits that process these distinct modalities are incompletely understood in any animal. We have systematically reconstructed, using light and electron microscopy, the synaptic targets of the photoreceptors specialized to detect color and polarized light in Drosophila. We identified known and novel downstream targets that are selective for different wavelengths as well as for polarized light and followed their projections to other areas in the optic lobes and the central brain. Strikingly, photoreceptors in the polarization-sensitive dorsal rim area target fewer cell types, that lack strong connections to the lobula, a neuropil with a proposed role in color processing. Our reconstruction identifies shared wiring and modality-specific specializations for color and polarization vision, and provides a comprehensive view of the first steps of the pathways processing color and polarized light inputs.