The novel pandemic called “Coronavirus Disease 2019” (COVID-19), as a global public health emergency and global threat, has affected many countries in unpredictable ways and impacted on physical activity (PA) behaviors to various extents. Specific populations including refugees, asylum seekers, and prisoners, are vulnerable groups with multiple complex health needs and worse health outcomes with respect to the general population worldwide and at high risk of death from the “Severe Acute Respiratory Syndrome-related Coronavirus type 2” (SARS-CoV-2). Governments around the world have been implementing preventive healthcare policies, including physical and social distancing, isolation, and confinement, to mitigate against the burden imposed by the COVID-19 outbreak. This pandemic period is characterized by reduced or lack of movement. During this period of lockdown, PA can represent an immunotherapy and a preventative approach to avoid the harmful effects of inactivity due to the pandemic. Moreover, PA could be prescribed to improve the immune system of specific populations (refugees, asylum seekers, and prisoners), which particularly experience the condition of being confined. The present narrative review discusses the potential impacts of COVID-19 pandemic on these specific populations’ health status and the importance of performing PA/exercise to reduce the deleterious effects of COVID-19 pandemic. In addition, we aim to provide useful recommendations on PA/exercise for these specific populations to maintain their level of independence, physical, and mental health as well as their wellbeing.