Mothers of children with autism spectrum disorder are more likely to have features of the broad autism phenotype and adverse mental health outcomes. The broad autism phenotype can contribute to poorer mental health; however, the mechanism behind this relationship remains uncertain. This study investigated both the predictors of mental health and the relationship between the broad autism phenotype and mental health symptoms in mothers of children with autism spectrum disorder. This cross-sectional survey recruited 232 Australian mothers (⩾18 years) of children with autism spectrum disorder (0–10 years) and assessed maternal broad autism phenotype, mental health, adult attachment style, degree of social support and parenting experience. Hierarchical multiple regressions, t-tests, correlations and mediation analyses were conducted. Psychological inflexibility was the greatest unique predictor of anxious, depressive and stress symptomatology. Challenging parenting experiences predicted depressive, anxious and stress symptomatology while the broad autism phenotype also uniquely predicted anxious symptoms. Both psychological inflexibility and challenging parenting experiences were significant mediators of the relationships between the broad autism phenotype and depressive symptoms and between the broad autism phenotype and anxiety. Psychological inflexibility alone mediated the relationship between the broad autism phenotype and stress. Targeting psychological inflexibility and supporting parenting are promising avenues for future interventions to support the mental health of mothers of young children with autism spectrum disorder. Lay abstract Mothers of children with autism tend to have poorer mental health outcomes compared to most mothers. Lack of social support, parenting challenges and relationship difficulties are more common for mothers of children with autism and can all affect maternal mental health. Mothers of children with autism are also more likely to have some autistic features, called the broad autism phenotype, that can contribute to poorer mental health; however, how these factors relate to one another are unclear. This study found that mothers who were less flexible in their thinking and behaviour and had more difficult parenting experiences tended to have poorer mental health. Mothers with more autistic features were less flexible in their thinking and behaviour, which, in turn, was linked to greater symptoms of depression, anxiety and stress. Mothers with greater autistic features also reported more difficult parenting experiences, which was in turn linked with greater symptoms of anxiety and depression. This study suggests that supporting mothers of young children with autism to manage parenting challenges and become more flexible with their thinking and behaviour could help to improve their mental health.