Background: The ongoing COVID-19 pandemic has created numerous stressful conditions, especially for vulnerable populations such as pregnant women. Pandemic-related pregnancy stress consists of two dimensions: stress associated with feeling unprepared for birth due to the pandemic (Preparedness Stress), and stress related to fears of perinatal COVID-19 infection (Perinatal Infection Stress). The purpose of our study was to elucidate the association between various factors—sociodemographic, obstetric, pandemic-related, and situational—and pandemic stress in its two dimensions during the second wave of the COVID-19 pandemic in Polish pregnant women. Methods: A cross-sectional study with a total of 1119 pregnant women recruited during the second wave of the COVID-19 pandemic in Poland (between November 2020 and January 2021). Participants were recruited via social media to complete an online study questionnaire that included sociodemographic, obstetric, situational, and COVID-19 pandemic factors, as well as the Pandemic-Related Pregnancy Stress Scale (PREPS). Results: Nearly 38.5% of participants reported high Preparedness Stress; 26% reported high Perinatal Infection Stress. Multivariate analyses indicated that lack of COVID-19 diagnosis, higher compliance with safety rules and restrictions, and limited access to outdoor space were independently associated with moderate to severe levels of Infection Stress. Current emotional or psychiatric problems, nulliparity, limited access to outdoor space, and alterations to obstetric visits were independently associated with moderate to severe Preparedness Stress. Conclusion: Study findings suggest that particular attention should be focused on the groups of pregnant women who are most vulnerable to pandemic-related stress and therefore may be more prone to adverse outcomes associated with prenatal stress.