Although primary caregivers of children with developmental disabilities (DDs) experience higher levels of distress than primary caregivers of typically developing children do, this problem has received limited attention in Greece. Therefore, this study examined mental health and associated factors among primary caregivers of children with and without DDs in Greece during the COVID-19 pandemic. This cross-sectional study included 156 participants. Primary caregivers completed a self-report survey on sociodemographic characteristics, the Depression, Anxiety, and Stress Scale-21 items, and the 12-item General Health Questionnaire. Primary caregivers of children with DDs, particularly mothers, reported more mental health problems and higher levels of psychological distress than the control group. Among families parenting a child with disabilities, caregivers’ psychological distress was significantly related to having a child with autism spectrum disorder and the severity of the child’s behavioral difficulties. Significant predictors of caregivers’ distress were the parent being female, the child being male, a single-parent family, a lower income, and higher depressive symptoms. Caregivers raising children with DDs face unique challenges in terms of care, necessitating the development of family-based interventions to improve the social-emotional well-being and overall quality of life for both parents and children.