Examining longitudinal data from the Toledo Adolescent Relationships Study (TARS) (n = 927), we assessed associations between physical victimization by an intimate partner, indicators of poor relationship quality, and depressive symptoms among young adult men and women in casually dating, exclusively dating, cohabiting, and marital relationships. In zero-order models, we found that physical victimization increased depressive symptoms. In multivariate models, victimization was a risk factor for depressive symptoms with the inclusion of prior depressive symptoms, family factors reflecting the intergenerational transmission of violence, sociodemographic background, and relationship characteristics including union status. Yet with the additional inclusion of indicators of poor relational quality, victimization was not a significant predictor of depressive symptoms. Arguing and poor communication influenced victimization and depressive symptoms. The associations between victimization and depressive symptoms did not differ by gender, nor were the effects of poor relationship quality on depressive symptoms conditional on gender. Thus, depressive symptoms are similarly responsive to intimate partner victimization, and for both women and men these associations were not significant with the inclusion of indicators of poor relationship quality. Findings underscored that victimization often occurs within relationship contexts characterized by a range of negative dynamics; thus multifaceted relationship-centered prevention and intervention efforts are likely to be more useful than those focusing only on negative messages about the use of aggression with an intimate partner.