According to a constructionist model of emotion, conceptual knowledge plays a foundational role in emotion perception; reduced availability of relevant conceptual knowledge should therefore impair emotion perception. Conceptual deficits can follow both degradation of semantic knowledge (e.g., semantic 'storage' deficits in semantic dementia) and deregulation of retrieval (e.g., semantic 'access' deficits in semantic aphasia). While emotion recognition deficits are known to accompany degraded conceptual knowledge, less is known about the impact of semantic access deficits. Here, we examined emotion perception and categorization tasks in patients with semantic aphasia, who have difficulty accessing semantic information in a flexible and controlled fashion following left hemisphere stroke. In Study 1, participants were asked to sort faces according to the emotion they portrayedwith numbers, written labels and picture examples each provided as category anchors across tasks.Semantic aphasia patients made more errors and showed a larger benefit from word anchors that reduced the need to internally constrain categorization than comparison participants.They successfully sorted portrayals that differed in valence (positive vs. negative) but had difficulty categorizing different negative emotions. They were unimpaired on a control task that involved sorting faces by identity. In Study 2, participants matched facial emotion portrayals to written labels following vocal emotion prosody cues, miscues, or no cues.Patients presented with overall poorer performance and benefited from cue trials relative to within-valence miscue trials. This same effect was seen in comparison participants, who also showed deleterious effects of within-valence miscue relative to no cue trials. Overall, we found that patients with deregulated semantic retrieval have deficits in emotional perception but that word anchors and cue conditions can facilitate emotion perception by increasing access to relevant emotion concepts and reducing reliance on semantic control. Semantic control may be of particular importance in emotion perception when it is necessary to interpret ambiguous inputs, or when there is interference between conceptually similar emotional states. These findings extend constructionist accounts of emotion to encompass difficulties in controlled semantic retrieval.