Color and polarization provide complementary information about the world and are detected by specialized photoreceptors. However, the downstream neural circuits that process these distinct modalities are incompletely understood in any animal. Using electron microscopy, we have systematically reconstructed the synaptic targets of the photoreceptors specialized to detect color and skylight polarization in Drosophila, and we have used light microscopy to confirm many of our findings. We identified known and novel downstream targets that are selective for different wavelengths or polarized light, and followed their projections to other areas in the optic lobes and the central brain. Our results revealed many synapses along the photoreceptor axons between brain regions, new pathways in the optic lobes, and spatially segregated projections to central brain regions. Strikingly, photoreceptors in the polarization-sensitive dorsal rim area target fewer cell types, and lack strong connections to the lobula, a neuropil involved in color processing. Our reconstruction identifies shared wiring and modality-specific specializations for color and polarization vision, and provides a comprehensive view of the first steps of the pathways processing color and polarized light inputs.