Children with Autism Spectrum Disorder (ASD) were particularly vulnerable to disruptions caused by the COVID-19 pandemic. In this study we conducted an anonymous caregiver survey to assess the impact of the COVID-19 pandemic on the psychosocial and behavioral health of children with ASD. Data from 700 responses identified several significant factors predicting greater difficulties for the child including pre-existing behavioral challenges (OR = 5.179; 95% CI: 2.696, 9.951), disrupted sleep (OR = 2.618; 95% CI 1.341, 5.112), and a diagnosis of depression (OR = 3.425; 95% CI: 1.1621, 4.116). Greater difficulties for caregivers in managing their child’s behaviors were associated with sleep disturbances (OR = 1.926; 95% CI: 1.170, 3.170), self-injurious behavior (OR = 3.587; 95% CI: 1.767, 7.281), and managing the child’s school activities (OR = 3.107; 95% CI: 1.732, 5.257) and free time (OR = 3.758; 95% CI: 2.217, 6.369). However, being under the care of a neuropsychiatrist was associated with less difficulty in managing the child’s behaviors (OR = 2.516; 95% CI: −1.046, −5.382). Finally, the presence of comorbidities (OR = 2.599; 95% CI: 1.053, 4.067) and a greater difficulty in managing the child’s school activities (OR = 2.531; 95% CI: 1.655, 3.868) and free time (OR = 1.651; 95% CI: 1.101, 2.478) were associated with an increased likelihood of caregiver desire for their child to return to in-person school in the fall. The COVID-19 pandemic had a wide-ranging impact on the behaviors of children with ASD and challenges for their caregivers.