The COVID-19 restrictions have impacted people’s lifestyles in all spheres (social, psychological, political, economic, and others). This study explored which factors affected the level of anxiety during the time of the first wave of COVID-19 and subsequent quarantine in a substantial proportion of 23 countries, included in this study. The data was collected from May to August 2020 (5 June 2020). The sample included 15,375 participants from 23 countries: (seven from Europe: Belarus, Bulgaria, Croatia, Hungary, Italy, Romania, Russia; 11 from West, South and Southeast Asia: Armenia, India, Indonesia, Iran, Iraq, Jordan, Malaysia, Pakistan, Saudi Arabia, Thailand, Turkey; two African: Nigeria and Tanzania; and three from North, South, and Central America: Brazil, Canada, United States). Level of anxiety was measured by means of the 7-item Generalized Anxiety Disorder Scale (GAD-7) and the 20-item first part of The State-Trait Anxiety Inventory (STAI)—State Anxiety Inventory (SAI). Respondents were also asked about their personal experiences with COVID-19, attitudes toward measures introduced by governments, changes in attitudes toward migrants during a pandemic, family income, isolation conditions, etc. The factor analysis revealed that four factors explained 45.08% of variance in increase of anxiety, and these components were interpreted as follows: (1) personal awareness of the threat of COVID-19, (2) personal reaction toward officially undertaken measures and attitudes to foreigners, (3) personal trust in official sources, (4) personal experience with COVID-19. Three out of four factors demonstrated strong associations with both scales of anxiety: high level of anxiety was significantly correlated with high level of personal awareness of the threat of COVID-19, low level of personal reaction toward officially undertaken measures and attitudes to foreigners, and high level of presence of personal experience with COVID-19. Our study revealed significant main effects of sex, country, and all four factors on the level of anxiety. It was demonstrated that countries with higher levels of anxiety assessed the real danger of a pandemic as higher, and had more personal experience with COVID-19. Respondents who trusted the government demonstrated lower levels of anxiety. Finally, foreigners were perceived as the cause of epidemic spread.