What is the function of auditory hemispheric asymmetry? We propose that the identification of sound sources relies on two complementary and perceptually relevant acoustic invariants — actions and objects — that are processed asymmetrically and sufficient to model and categorize any sound. We observed that environmental sounds are an independent combination of purely temporal and spectral acoustic modulations. Behaviorally, discrimination of actions relied on temporal modulations, while discrimination of objects relied on spectral modulations. Functional magnetic resonance imaging data showed that actions and objects are respectively decoded in the left and right hemispheres, in bilateral superior temporal and left inferior frontal regions. This asymmetry reflects a generic differential processing — through differential neural sensitivity to temporal and spectral modulations present in all environmental sounds — that supports the efficient categorization of actions and objects. These results provide an ecologically valid framework of the functional role of auditory brain asymmetry.