Background: Although numerous studies have assessed physical activity during pregnancy and relationships with infant outcomes, such as birthweight, few have evaluated sedentary behavior. Our objective was to evaluate sedentary behavior across pregnancy and relationships with infant birthweight in a sociodemographically diverse sample. Methods: We measured device-assessed sedentary behavior and physical activity over three days at 16–18, 24–26, and 32–34 weeks gestation and infant birthweight from medical records among 71 participants. We used linear regression to assess relationships between sedentary behavior at each evaluation period with birthweight-for-gestational age Z-scores (BW-for-GA). Results: There were no linear relationships between sedentary behavior and BW-for-GA at any evaluation period. We observed a modest curvilinear relationship between sedentary behavior at 16–18 weeks and BW-for-GA (R2 = 0.073, p = 0.021). Low and high levels of sedentary behavior predicted lower BW-for-GA. Multivariate models suggested that this relationship was independent of physical activity levels. Conclusions: Considering the high levels of sedentary behavior during pregnancy observed in many studies, even modest associations with birthweight merit further consideration. Relationships might not be evident later in pregnancy or if only linear relationships are considered. More detailed studies could help guide recommendations on sedentary behavior during pregnancy and the development of more comprehensive interventions.