Background: Efficient neural encoding of sound plays a critical and widespread role in speech and language, and when impaired, may have reverberating effects on a range of communication skills. Autism spectrum disorder (ASD) is a condition involving impaired communication abilities, including atypical prosody (e.g., intonation modulation, rate, rhythm), in which atypical neural processing of speech has been implicated. Parallel patterns of communication differences have been noted in parents of individuals with ASD, who may exhibit subclinical language-related differences believed to reflect genetic liability to ASD. The present study investigated disruptions to neural processing of temporal and spectral (i.e., frequency/pitch) properties of speech sounds as a potential neurobiological mechanism underlying specific language-related impairments in ASD and related subclinical differences documented among first-degree relatives of individuals with ASD. Methods: Participants included individuals with ASD, their parents, and respective control groups. Group differences in temporal and spectral processing based on the neural FFR were assessed using MANCOVAs controlling for chronological age. Relationships between the FFR, pragmatic language ability, and receptive and expressive prosody skills were assessed using Pearson correlations. Familiality of the FFR in mother-child dyads were also examined using Pearson correlations.Results: Findings revealed inefficiencies in neural encoding of speech sounds in both individuals with ASD and their parents (specifically, increased neural noise and delayed neural processing of speech sounds), as well as a less robust neural representation of spectral properties of speech sounds in individuals with ASD. Associations between neural processing of speech sounds and language-related abilities were detected in both groups, along with evidence of familiality of neural pitch processing in the ASD family groups. Limitations: Additional investigation is necessary to determine whether relationships between FFR and pragmatic language and prosody extend to individuals with more severe language and/or cognitive impairments. Conclusions: Overall, results suggest atypical neural processing of speech sounds may constitute an important, heritable ingredient contributing to the ASD language phenotype and subclinical phenotypic expression of genetic liability in parents.