Following the serious health situation in Spain and around the world in 2020 and 2021 stemming from COVID-19, this paper examines how the most vulnerable groups (in social and health terms) in Spanish society suffered the worst impacts on their quality of life during the pandemic. The review of the literature and publications released by the health authorities highlight the high number of cases of illness and death due to COVID-19; however, there are no studies about how it has affected aspects of citizens’ daily lives, considering some of their sociodemographic characteristics such as age and gender. A key issue when measuring citizens’ quality of life is that we typically see a U-shaped curve by age group, where young people register the highest shares along with the elderly; nevertheless, the pandemic has clearly brought about a change in this regard. In this study, we corroborate this finding, along with the psychological issues underlying this fact in young people and the physical ones in the elderly. To do so, we use an ANOVA and regression analysis with data from a sample of 908 Spanish citizens belonging to different age groups. These data were collected through an online survey distributed throughout Spain, mostly by email and social media, between 15 February and 15 April 2021 with a margin of error of 3.25% and a confidence level of 95%. Furthermore, the analysis allowed us to determine how aspects of daily life—family situation, living conditions, social environment, employment and financial situation—have been negatively affected during the pandemic, clearly depending on the age of the people surveyed, pushing them towards social and health vulnerability.