Rare copy-number variants (CNVs) and gene-disruptive mutations associated with neurodevelopmental disease are characterized by phenotypic heterogeneity. When affected children inherit these mutations, they usually present more severe features than carrier parents, leading to challenges in diagnosis and management. To understand how the genetic background modulates phenotypes of these variants, we analyzed clinical and exome-sequencing data from 757 probands and 233 parents and siblings who carry disease-associated mutations. We found that the number of rare pathogenic secondary mutations in developmental genes (second-hits) modulates the expressivity of disease in probands with 16p12.1 deletion (n=26, p=0.014) and in autism probands with gene-disruptive mutations (n=184, p=0.031) when compared to their carrier family members. Probands with 16p12.1 deletion and a strong family history of neuropsychiatric disease were more likely to manifest multiple and more severe clinical features (p=0.035) and carry a higher burden of second-hits compared to those with mild or no family history (p=0.001). The amount of secondary variants determined the severity of cognitive impairment in 432 probands carrying pathogenic rare CNVs or de novo mutations in disease genes and negatively correlated with head size in 84 probands with 16p11.2 deletion, suggesting an effect of the genetic background across multiple phenotypic domains. These second-hits involved known disease genes, such as SETD5, AUTS2, and NRXN1, and novel candidate modifiers, such as CDH23, RYR3, and DNAH3, affecting core developmental processes. Our findings suggest that in the era of personalized medicine, accurate diagnosis will require complete evaluation of the genetic background even after a candidate gene mutation is identified.