Physical activity (PA) is safe for most pregnant women, improving both maternal fitness and birth outcomes. Despite evidence of benefits, most pregnant women eliminate or reduce PA during pregnancy. This systematic review aimed to analyze the factors affecting maternal PA during pregnancy with reference to a socio-ecological model. A systematic search of relevant published studies between 2001 and 2022 was conducted through PubMed, Scopus, Web of Science, Academic Search Ultimate, Medline, and SPORTDiscus with full text via the EBSCO platform. A total of 32 studies that met the inclusion criteria were reviewed. The findings revealed that various study designs can lead to different outcomes in terms of what is identified as a PA facilitator or barrier. The factors that positively influenced PA in pregnant women were: higher levels of education, knowledge, and skills, as well as access to mass media. Conversely, lower levels of education, lack of knowledge and skills, low income, pregnancy discomforts, limited time, safety concerns, and societal perceptions of PA in pregnancy acted as barriers. Additionally, family, colleagues/friends, and partners could either support or hinder PA. Factors affecting overall maternal PA were somewhat different from those affecting the moderate-to-vigorous intensity of PA. Pregnant women receive little organizational and policy support. There is an urgent need to provide accessible information and resource systems for pregnant women. Since most pregnant women are motivated to engage in PA and susceptible to family advice, interventions should not be limited only to pregnant women, but should involve a family member, especially partners. There is a need to take global, systemic actions to promote an active lifestyle in pregnancy. Addressing safety concerns related to PA during pregnancy should be a significant part of these promotional activities.