In the United States, college students have been identified as a high-risk group for sexual violence (SV) victimization and perpetration, with one in three women and one in four men experiencing SV in their lifetimes. Therefore, studies investigating both perpetration and victimization across genders are pertinent. Additionally, in the context of higher education, athletics is often viewed as a high-risk environment for SV. Despite established relationships between SV and intercollegiate athletics, research on this topic is not conclusive due to the heterogeneous nature of college sports, as well as a dearth of research on recreational athletes. The primary purpose of this study was to examine the differences in SV victimization and perpetration among intercollegiate, recreational, and non-athlete populations. Additional context was provided by exploring variables at multiple levels of the socio-ecological model for violence prevention, as these factors can be important aspects of a person’s development and may inform behavior. Utilizing a sample of 62,996 students from 199 institutions of higher education in the United States, multilevel logistic regression analyses were conducted to assess individual- (i.e., personal demographics), relationship- (i.e., athlete status), and community-level (i.e., institutional characteristics) predictors of SV victimization and perpetration. The results indicated that recreational athletes are the most at-risk group, experiencing and perpetrating SV at higher rates than non-athletes. Results on institutional status and individual characteristics, as well as implications for practitioners and future research is discussed.