Increased life expectancy in Spain has highlighted the need for formal caregivers who care for older adults who live at home. In many cases, families choose to hire immigrants from Latin America who, on many occasions, have been forced to emigrate, which causes a considerable psychological impact on them. To this, other factors such as precarious working conditions, loneliness, or high workload are added, which leads to many caregivers becoming sick. The aim of this study was to assess the quality of life and the level of anxiety derived from the workload of Latin American immigrants who come to Spain to care for older adults. A descriptive cross-sectional study was carried out using two validated questionnaires to measure this relationship. A correlation was also established between quality of life and anxiety as expressed by the participants. The sample consisted of a total of 426 subjects. One of the main results showed that the lower the caregiver’s quality of life, the higher their level of anxiety (OR = 1.06; 95% CI). Live-in caregivers had a lower quality of life than people who did not live in the same house as the cared-for person (OR = 2.22; 95% CI). Working long hours and having a high workload was associated with poorer quality of life and higher levels of anxiety. Assessing immigrants who are formal caregivers and providing them with the support that helps to improve their quality of life is suggested to reduce the occurrence of anxiety disorders.