Numerous studies indicate that deficits in the proper integration or migration of specific GABAergic precursor cells from the subpallium to the cortex can lead to severe cognitive dysfunctions and neurodevelopmental pathogenesis linked to intellectual disabilities. A different set of GABAergic precursors cells that express Pax2 migrate to hindbrain regions, targeting, for example auditory or somatosensory brainstem regions. We demonstrate that the absence of BDNF in Pax2-lineage descendants of BdnfPax2KOs causes severe cognitive disabilities. In BdnfPax2KOs, a normal number of parvalbumin-positive interneurons (PV-INs) was found in the auditory cortex (AC) and hippocampal regions, which went hand in hand with reduced PV-labeling in neuropil domains and elevated activity-regulated cytoskeleton-associated protein (Arc/Arg3.1; here: Arc) levels in pyramidal neurons in these same regions. This immaturity in the inhibitory/excitatory balance of the AC and hippocampus was accompanied by elevated LTP, reduced (sound-induced) LTP/LTD adjustment, impaired learning, elevated anxiety, and deficits in social behavior, overall representing an autistic-like phenotype. Reduced tonic inhibitory strength and elevated spontaneous firing rates in dorsal cochlear nucleus (DCN) brainstem neurons in otherwise nearly normal hearing BdnfPax2KOs suggests that diminished fine-grained auditory-specific brainstem activity has hampered activity-driven integration of inhibitory networks of the AC in functional (hippocampal) circuits. This leads to an inability to scale hippocampal post-synapses during LTP/LTD plasticity. BDNF in Pax2-lineage descendants in lower brain regions should thus be considered as a novel candidate for contributing to the development of brain disorders, including autism.