Most governments have enacted physical or social distancing measures to control COVID-19 transmission. Yet little is known about the socio-economic trade-offs of these measures, especially for vulnerable populations, who are exposed to increased risks and are susceptible to adverse health outcomes. To examine the impacts of physical distancing measures on the most vulnerable in society, this scoping review screened 39,816 records and synthesised results from 265 studies worldwide documenting the negative impacts of physical distancing on older people, children/students, low-income populations, migrant workers, people in prison, people with disabilities, sex workers, victims of domestic violence, refugees, ethnic minorities, and people from sexual and gender minorities. We show that prolonged loneliness, mental distress, unemployment, income loss, food insecurity, widened inequality and disruption of access to social support and health services were unintended consequences of physical distancing that impacted these vulnerable groups and highlight that physical distancing measures exacerbated the vulnerabilities of different vulnerable populations.