The emotional properties of music are influenced by a host of factors, such as timbre, mode, harmony, and tempo. In this paper, we consider how two of these factors, mode (major vs. minor) and timbre interact to influence ratings of perceived valence, reaction time, and recognition memory. More specifically, we considered the notion of congruence—that is, we used a set of melodies that crossed modes typically perceived as happy and sad (i.e., major and minor) in Western cultures with instruments typically perceived as happy and sad (i.e., marimba and viola). In a reaction-time experiment, participants were asked to classify melodies as happy or sad as quickly as possible. There was a clear congruency effect—that is, when the mode and timbre were congruent (major/marimba or minor/viola), reaction times were shorter than when the mode and timbre were incongruent (major/viola or minor/marimba). In Experiment 2, participants first rated the melodies for valence, before completing a recognition task. Melodies that were initially presented in incongruent conditions in the rating task were subsequently recognized better in the recognition task. The recognition advantage for melodies presented in incongruent conditions is discussed in the context of desirable difficulty.