The role of subcortical structures in binaural integration is of great interest for auditory processing. The inferior colliculus (IC) is a main auditory midbrain center where ascending and descending auditory projections converge, which was suggested to encode auditory information via a push-pull mechanism between the two ICs. However, the origin of this push-pull mechanism in the brain and how it interacts with other upstream/downstream subcortical areas remain to be elucidated. Here, we harness functional MRI (fMRI) in combination with IC lesions in the rat to dissect the push-pull interaction from a brain-wide perspective. We find evidence for the push-pull mechanism in IC through negative/positive fMRI signals in the ipsilateral/contralateral ICs upon monaural stimulation. By unilaterally lesioning the corresponding contralateral IC, we demonstrate the necessity of intercollicular interactions for the push-pull interaction. Using binaural stimulation and IC lesions, we show that the push-pull interaction is exerted also in binaural processing. Finally, we demonstrate that, at least at the population level revealed by fMRI, the main push-pull interactions occur first at the IC level, and not earlier, and that the outcome of the push-pull "calculation" is relayed downstream to MGB. This dissection of the push-pull interaction sheds light into subcortical auditory function.