Social workers in the community setting are in constant contact with the suffering experienced by the most vulnerable individual. Social interventions are complex and affect social workers’ emotional well-being. The aim of this study was to identify the emotions, feelings, and experiences social workers have while attending to individuals in situations of vulnerability and hardship. A qualitative methodology based on hermeneutic phenomenology was used. Six interviews and two focus group sessions were conducted with social workers from the community social services and health services of the Andalusian Public Health System in the province of Almería (Spain). Atlas.ti 8.0 software was used for discourse analysis. The professionals highlighted the vulnerability of certain groups, such as the elderly and minors, people with serious mental problems, and people with scarce or no economic resources. Daily contact with situations of suffering generates a variety of feelings and emotions (anger, sadness, fear, concern). Therefore, more attention should be paid to working with the emotions of social workers who are exposed to tense and threatening situations. Peer support, talking, and discussions of experiences are pointed out as relevant by all social workers. Receiving training and support (in formal settings) in order to learn how to deal with vulnerable groups could be positive for their work and their professional and personal quality of life.