The coronavirus pandemic and related government restrictions have a significant impact on peoples’ everyday functioning and working, which influences their physical and mental health. The aim of the study was to examine the associations between stress and sleep quality of people of different working modes: working in the workplace (WP), working remotely (RW), and nonworking (NW) in relation to their physical activity (PA) during COVID-19 pandemic lockdown in Poland. It was an online survey performed during governmental lockdown in April 2020. The data were collected form 1959 adults using International Physical Activity Questionnaire—Short Form (IPAQ-SF), Pittsburgh Sleep Quality Index (PSQI), and Perceived Stress Scale (PSS). The conducted analysis included t-Student test, analysis of variance (ANOVA), and mediation analysis (MANOVA). A moderate level of stress was reported in 57% of participants, and 34% of them reported a high stress level. Poor sleep quality was reported in 64% of participants. Total PA performed daily was, on average, 184.8 ± 170.5 min/day for WP, 120.6 ± 124.4 min/day for RW, and 124.6 ± 114.7 min/day for NW (p < 0.001). There was a relationship observed between the stress and sleep quality vs. PA habit and working mode, with p < 0.05. Being physically active can be beneficial to perceive less stress and sleep disturbances influencing sleep quality, especially in remotely or nonworking people. Planning future pandemic restrictions, the policymakers should be aware of the appropriate guidelines of work planning and PA recommendations for people of different working modes.