With the introduction of the ICD-11 and DSM-5, indicators of adaptive behavior, including social–emotional skills, are in focus for a more comprehensive understanding of neurodevelopmental disorders. Emotional skills can be assessed with the Scale of Emotional Development-Short (SED-S). To date, little is known about the effects of physical disorders and sensory impairments on a person’s developmental trajectory. The SED-S was applied in 724 adults with intellectual disabilities, of whom 246 persons had an additional physical and/or sensory impairment. Ordinal regression analyses revealed an association of movement disorders with more severe intellectual disability and lower levels of emotional development (ED) on the overall and domain levels (Others, Body, Material, and Communication). Visual impairments predicted lower levels of ED in the SED-S domains Material and Body, but not the overall level of ED. Hearing impairments were not associated with intellectual disability or ED. Epilepsy correlated only with the severity of intellectual disability. Multiple impairments predicted more severe intellectual disabilities and lower levels of overall ED. In conclusion, physical and sensory impairments may not only affect physical development but may also compromise intellectual and emotional development, which should be addressed in early interventions.