Background: Intimate Partner Violence (IPV) causes physical, sexual, or psychological harm. The association between psychosexual (sexual assertiveness, erotophilia, and attitude towards sexual fantasies) and sexual function (sexual desire, sexual excitation, erection, orgasm capacity, and sexual satisfaction), and the experience of physical and non-physical IPV was assessed. Methods: Data from 3394 (1766 women, 1628 men) heterosexual adults completed the Spanish version of the Index of Spouse Abuse, scales measuring psychosexual and sexual function, and demographic characteristics were collected. Results: For men, poorer sexual health was associated with an experience of physical abuse (F = 4.41, p < 0.001) and non-physical abuse (F = 4.35, p < 0.001). For women, poorer sexual health was associated with physical abuse (F = 13.38, p < 0.001) and non-physical abuse (F = 7.83, p < 0.001). Conclusion: The experience of physical or non-physical abuse has a negative association with psychosexual and sexual functioning in both men and women.