Endometriosis is a disabling disease that may significantly compromise a woman’s social relationships, sexuality, and mental health. Considering the impact of endometriomas and deep infiltrating endometriosis (DIE) on quality of life and the limited number of papers on this topic, the objective of this study was to assess health-related quality of life (HRQoL) in adult Spanish women with the condition. A case-control study was conducted on a group of 99 patients with ovarian endometriomas or DIE and 157 controls. Women underwent physical and gynecological examinations, and they completed health questionnaires including the Short Form-12v2 (SF-12v2), a survey for HRQoL. Eight scales and two component summary scores (Physical (PCS) and Mental (MCS), respectively) were calculated. Women with endometriomas or DIE had significantly worse PCS: 47.7 ± 9.7 vs. 56.1 ± 5.9, respectively (p < 0.001) compared to controls, as well as lower scores on seven out of the eight scales (p < 0.01). No significant differences were found for the MCS. Conclusions: HRQoL was significantly lower in patients with endometriomas or DIE compared to controls. If confirmed, these results may have important implications for prevention, clinical practice, and intervention.