Epidemiological data show that human reproductive disorders are a common problem worldwide, affecting almost one in six people of reproductive age. As a result, infertility has been identified by the World Health Organization as a public health disease. Reproductive problems can take a heavy toll on the psychosocial well-being of couples suffering from infertility. This is especially true for women, who tend to be the ones who undergo the most treatment. The main objective of the present study is to find out whether a sex-based infertility diagnosis influences the quality of life of couples with infertility. Also, we aim to find out whether the degree of adherence to gender norms influences their quality of life. A cross-sectional study was conducted using the Fertility Quality of Life Questionnaire (FertiQoL) and the Conformity to Feminine and Masculine Norms Inventories in a sample of 219 infertile Spanish couples (438 participants). The results show that, in all cases, regardless of the degree of conformity to gender norms and whether the infertility diagnosis was of female or male origin, women have lower scores on the self-perceived quality of life. This suggests that being female is already a psychosocial risk factor when assessing the psychosocial consequences of infertility.