Intimate partner sexual violence (IPSV) is a common yet hidden form of violence against women. It encompasses a range of behaviors, including rape and sexual assault, and also more subtle behaviors such as the use of coercion and blackmail to obtain sex. It is typically described as an aspect of intimate partner violence, yet, although it often co-occurs with physical or psychological abuse, the contextual factors and nuances of sexual violence perpetrated by an intimate partner are likely to be very different. IPSV also differs greatly from sexual assault perpetrated by a stranger or other known person. Despite this, ecological theories and models developed to help understand and prevent violence against women have neglected or excluded IPSV. This is problematic given the serious mental and physical health problems consistently associated with it. In response, this article aims to explore the ecological factors that may be associated with IPSV at the individual, relationship, community, and societal levels. It draws on both inductive and deductive thematic analysis of N = 38 in-depth, unstructured interviews with women victim/survivors. Individual-level factors included sexual inexperience and past trauma (for victims), and fragile masculinity and a sense of entitlement to sex (for perpetrators). Relationship-level factors included a large age gap in the relationship and co-occurring psychological abuse. Community-level factors were identified as failure to talk about sex, isolation, and lack of support, while societal-level factors included the idea that sex is a woman’s duty in a relationship, and that “real rape” is perpetrated by strangers. These factors are incorporated into a proposed ecological model that is unique to IPSV, furthering our understanding of this complex problem and its areas of overlap and difference with intimate partner violence and sexual violence. With refinement and testing through large-scale quantitative studies, this model may be critical in guiding future prevention efforts.