While large numbers of women report high levels of psychological distress associated with endometriosis, others report levels of distress that are comparable to those of healthy women. Thus, the aim of the current study was to develop an explanatory model for the effect of endometriosis on women’s psychological distress. Furthermore, it sought to further investigate the role of body image, self-criticism, and pain intensity on the psychological distress associated with endometriosis and establish the effect of chronic illness load on the development of this distress. This study comprised a total of 247 women aged 20–49 (M = 31.3, SD = 6.4)—73 suffering from endometriosis only, 62 suffering from endometriosis and an additional chronical illness (ACI), and 112 healthy peers (HP)—who completed the Patient Health Questionnaire, the Generalized Anxiety Disorder-Item Scale, the Body Appreciation Scale-2, and the Self-Criticism Sub-Scale. When comparing each endometriosis group to their HP’s, we found that the differences between HP and endometriosis ACI in depression and anxiety were mediated by body image (Betas = 0.17 and 0.09, respectively, p’s < 0.05) and self-criticism (Betas = 0.23 and 0.26, respectively, p’s < 0.05). When comparing endometriosis participants to endometriosis ACI participants, differences in depression were mediated by body image, self-criticism, and pain intensity (Betas = 0.12, 0.13, 0.13 respectively, p’s < 0.05), and the differences in anxiety were mediated by self-criticism and pain intensity (Betas = 0.19, 0.08, respectively, p’s < 0.05). Physicians and other health professionals are advised to detect women with endometriosis ACI who are distressed, and to offer them appropriate intervention.