Face-to-face communication requires skills that go beyond core language abilities. In dialog, we routinely make inferences beyond the literal meaning of utterances and distinguish between different speech acts based on e.g. contextual cues. It is however not known whether such communicative skills potentially overlap with core language skills or other capacities, such as Theory of Mind (ToM). In this fMRI study we investigate these questions by capitalizing on individual variation in pragmatic skills in the general population. Based on behavioral data from 201 participants, we selected participants with higher vs lower pragmatic skills for the fMRI-study (N = 57). In the scanner, participants listened to dialogs including a direct or an indirect target utterance. The paradigm allowed participants at the whole group level to (passively) distinguish indirect from direct speech acts, as evidenced by a robust activity difference between these speech acts in an extended language network including ToM areas. Individual differences in pragmatic skills modulated activation in two additional regions outside the core language regions (one cluster in the left lateral parietal cortex and intraparietal sulcus and one in the precuneus). The behavioral results indicate segregation of pragmatic skill from core language and ToM. In conclusion, contextualized and multimodal communication requires a set of inter-related pragmatic processes that are neurocognitively segregated: (1) from core language and (2) partly from ToM.