Breast cancer survivorship is a recognized risk factor for sexual dysfunction, with various clinical, sociocultural, and psychological factors potentially interacting differently across populations. This study compared sexual dysfunction, anxiety, and depression between females with breast cancer and those without, aiming to identify associated factors. A total of 362 females participated, including 227 with sexual dysfunction and 135 controls. Among them, 195 are breast cancer survivors, while 167 have no personal history of cancer. Key variables were analyzed using Student’s t-test for quantitative data and Fisher’s exact test for categorical data, while logistic regression models were used to assess the association between sexual dysfunction and various factors. Multivariate analysis revealed that, in sexually active females, breast cancer survivorship increased the odds of sexual dysfunction 2.7-fold (95% CI: 1.17–6.49; p = 0.020). Anxiety was significantly associated with sexual dysfunction, regardless of cancer status (AdOR 6.00; 95% CI: 2.50–14.43; p < 0.001). The interaction between cancer survival and anxiety further increased the odds of sexual dysfunction by more than 11-fold (AdOR 11.55; 95% CI: 3.81–35.04; p < 0.001). Additionally, obesity was found to be a protective factor among cancer survivors (AdOR 0.149; 95% CI: 0.027–0.819; p = 0.029). In conclusion, breast cancer has a significant impact on sexual function, with psychological factors like anxiety playing a crucial role. Addressing these issues requires a holistic, patient-centered approach that considers the complex interplay of physical, emotional, and sociocultural factors.