Due to the ongoing COVID-19 pandemic, an unprecedented number of people worldwide is currently affected by quarantine or isolation. These measures have been suggested to negatively impact on mental health. We conducted the first systematic literature review and meta-analysis assessing the psychological effects in both quarantined and isolated persons compared to non-quarantined and non-isolated persons. PubMed, PsycINFO, and Embase databases were searched for studies until April 22, 2020 (Prospero Registration-No.: CRD42020180043). We followed PRISMA and MOOSE guidelines for data extraction and synthesis and the Newcastle–Ottawa Scale for assessing risk of bias of included studies. A random-effects model was implemented to pool effect sizes of included studies. The primary outcomes were depression, anxiety, and stress-related disorders. All other psychological parameters, such as anger, were reported as secondary outcomes. Out of 6807 screened articles, 25 studies were included in our analyses. Compared to controls, individuals experiencing isolation or quarantine were at increased risk for adverse mental health outcomes, particularly after containment duration of 1 week or longer. Effect sizes were summarized for depressive disorders (odds ratio 2.795; 95% CI 1.467–5.324), anxiety disorders (odds ratio 2.0; 95% CI 0.883–4.527), and stress-related disorders (odds ratio 2.742; 95% CI 1.496–5.027). Among secondary outcomes, elevated levels of anger were reported most consistently. There is compelling evidence for adverse mental health effects of isolation and quarantine, in particular depression, anxiety, stress-related disorders, and anger. Reported determinants can help identify populations at risk and our findings may serve as an evidence-base for prevention and management strategies.