The phenomenon of musical 'consonance' is crucial for many musical styles, determining how notes are organized into scales, how scales are tuned, and how chords are constructed from scales. Western music theory assumes that consonance depends solely on frequency ratios between chord tones; however, psychoacoustic theories predict a dependency also on the 'timbre' (tone color) of the underlying sounds. We investigate this possibility with 24 large-scale behavioral experiments (4,666 participants), constructing detailed continuous maps of consonance judgments for different timbres, and simulating these judgments with representative computational models. We find that timbral manipulations can indeed modify consonance judgments, transforming both the magnitude and the location of consonance peaks. We show how these results shed new light into the mechanisms underlying consonance perception as well as the cultural evolution of scale systems. More broadly, this work showcases how large-scale behavioral experiments can inform classical questions in auditory perception.