Verbal irony conveys various emotional messages, from criticism to humor, that differ from the meaning of the actual words. To understand irony, we need conceptual knowledge of irony in addition to an understanding of context. We investigated the neural mechanism of irony comprehension, focusing on two overlooked issues: conceptual knowledge and emotional response. We studied 35 healthy subjects who underwent functional MRI. During the scan, the subject examined first-person-view stories describing verbal interactions, some of which included irony directed toward the subject. After MRI, the subject viewed the stories again and rated the degree of irony, humor, and negative emotion evoked by the statements. We identified several key findings about irony comprehension: (1) the right anterior superior temporal gyrus may be responsible for representing social conceptual knowledge of irony, (2) activation in the medial prefrontal cortex and the right anterior inferior temporal gyrus might underlie the understanding of context, (3) modulation of activity in the right amygdala, hippocampus, and parahippocampal gyrus is associated with the degree of irony perceived, and (4) modulation of activity in the right dorsolateral prefrontal cortex varies with the degree of humor perceived. Our results clarified the differential contributions of the neural loci of irony comprehension, enriching our understanding of pragmatic language communication from a social behavior point of view.