Sound localization requires rapid interpretation of signal speed, intensity, and frequency. Precise neurotransmission of auditory signals relies on specialized auditory brainstem synapses including the calyx of Held, the large encapsulating input to principal neurons in the medial nucleus of the trapezoid body (MNTB). During development, synapses in the MNTB are established, eliminated, and strengthened, thereby forming an excitatory/inhibitory (E/I) synapse profile. However, in neurodevelopmental disorders such as autism spectrum disorder (ASD), E/I neurotransmission is altered, and auditory phenotypes emerge anatomically, molecularly, and functionally. Here we review factors required for normal synapse development in this auditory brainstem pathway and discuss how it is affected by mutations in ASD-linked genes.