The highly conserved Elongator complex is a translational regulator that plays a critical role in neurodevelopment, neurological diseases, and brain tumors. Numerous clinically relevant variants have been reported in the catalytic Elp123 subcomplex, while no missense mutations in the accessory subcomplex Elp456 have been described. Here, we identify ELP4 and ELP6 variants in patients with developmental delay, epilepsy, intellectual disability, and motor dysfunction. We determine the structures of human and murine Elp456 subcomplexes and locate the mutated residues. We show that patient-derived mutations in Elp456 affect the tRNA modification activity of Elongator in vitro as well as in human and murine cells. Modeling the pathogenic variants in mice recapitulates the clinical features of the patients and reveals neuropathology that differs from the one caused by previously characterized Elp123 mutations. Our study demonstrates a direct correlation between Elp4 and Elp6 mutations, reduced Elongator activity, and neurological defects. Foremost, our data indicate previously unrecognized differences of the Elp123 and Elp456 subcomplexes for individual tRNA species, in different cell types and in different key steps during the neurodevelopment of higher organisms.