Early loss of a given sensory input in mammals causes anatomical and functional modifications in the brain via a process called cross-modal plasticity. In the past four decades, several animal models have illuminated our understanding of the biological substrates involved in cross-modal plasticity. Progressively, studies are now starting to emphasise on cell-specific mechanisms that may be responsible for this intermodal sensory plasticity. Inhibitory interneurons expressing γ-aminobutyric acid (GABA) play an important role in maintaining the appropriate dynamic range of cortical excitation, in critical periods of developmental plasticity, in receptive field refinement, and in treatment of sensory information reaching the cerebral cortex. The diverse interneuron population is very sensitive to sensory experience during development. GABAergic neurons are therefore well suited to act as a gate for mediating cross-modal plasticity. This paper attempts to highlight the links between early sensory deprivation, cortical GABAergic interneuron alterations, and cross-modal plasticity, discuss its implications, and further provide insights for future research in the field.