Adolescents develop within multiple contexts that synergistically influence their behavior and health. To understand the simultaneous influence of neighborhood and family contexts on adolescents, this study examined relationships of neighborhood socioeconomic disadvantage, neighborhood social disorganization, family conflict, parent-child bonding and parental control with trajectories of physical and social aggression. The sample included 5,118 adolescents between ages 11 and 18 (50% female, 52% Caucasian) living in predominantly rural areas. Multilevel growth curve models showed an interaction between neighborhood disadvantage, family conflict and gender on the physical aggression trajectories. The interaction suggested more rapid processes of both increase in and desistance from physical aggression over time for boys with high neighborhood disadvantage and high family conflict, as well as a higher starting point, more gradual increase and slower process of desistance over time for girls in similar neighborhood and family contexts. Less parent-child bonding and less parental control also were associated with higher initial levels of physical aggression. For social aggression, an interaction between family conflict and gender showed girls with high family conflict had the highest initial levels of social aggression, with a more gradual increase over time for these girls compared to their male counterparts in high-conflict families or their female counterparts in low-conflict families. Less parent-child bonding was associated with higher initial levels and a faster increase over time of social aggression, and less parental control was associated with higher initial levels of social aggression. The findings suggest early family-based interventions may help prevent perpetration of both physical and social aggression during adolescence.