Background and objectives: The COVID-19 pandemic has had an unprecedented reliance on informal caregivers as one of the pillars of healthcare systems. The aim of this study was to assess the quality of life of informal caregivers during the COVID-19 epidemic in Serbia. Materials and Methods: A cross-sectional study was conducted among informal caregivers during the COVID-19 epidemic in Serbia. Physical and mental quality of life was measured by the 36-Item Short-Form Health Survey. Additional data included sociodemographic characteristics, caregiver and care recipient characteristics, and COVID-19 related concerns. The qualitative component was performed using focus groups and individual in-depth interviews. Results: Out of 112 informal caregivers enrolled, most were female (80%), and the average age was 51.1 ± 12.3 years. The majority was delivering care to one person, who was a family member, on a daily basis (86.4%, 92%, and 91.1%, respectively). In multiple regression models, significant predictors of caregivers’ physical health were delivering care to a family member and a higher level of care complexity, while significant predictors of caregivers’ mental health were a higher level of care complexity and increased concerns about self-health and the health of the person being cared for due to the COVID-19 epidemic. Conclusions: Informal caregivers are experiencing negative physical and mental health outcomes during the COVID-19 epidemic in Serbia.