During the COVID-19 outbreak in March 2020, restrictive measures (e.g., prohibiting physical visits and group activities) were introduced in nursing homes to protect older residents. Although the importance of social contacts and social activities to fulfill social needs and avoid loneliness is known, these were challenged during the pandemic. This qualitative study specifically focused on how residents, close relatives, and volunteers in nursing homes experienced the restrictive measures in retrospect and gained insights into the impact of the restrictive measures on social needs and loneliness, and the lessons that could be learned. Thirty semi-structured, face-to-face interviews with residents and close relatives, and one online focus group with ten volunteers, were conducted. Recruitment took place at psychogeriatric and somatic units in the Northern, Eastern and Southern regions of the Netherlands and Flanders, Belgium. The interviews and focus group were transcribed verbatim, and an open, inductive approach was used for analysis. Alternative ways of social contact could not fully compensate for physical visits. Generally, participants reported that it was a difficult time, indicated by feelings of loneliness, fear, sadness, and powerlessness. A great diversity in loneliness was reported. The most important reasons for feeling lonely were missing close social contacts and social activities. The diversity in the impact of restrictive measures depended on, e.g., social needs, coping strategies, and character. Restrictive COVID-19 measures in nursing homes resulted in negative emotions and unmet social needs of residents, close relatives, and volunteers. During future outbreaks of the COVID-19 virus or another virus or bacterium, for which restrictive measures may be needed, nursing homes should actively involve residents, close relatives, and volunteers to balance safety, self-determination, and well-being.