The primary function of language is to communicate—that is, to make individuals reach a state of mutual understanding about a particular thought or idea. Accordingly, daily communication is truly a task of social coordination. Indeed, successful interactions require individuals to (1) track and adopt a partner’s perspective and (2) continuously shift between the numerous elements relevant to the exchange. Here, we use a referential communication task to study the contributions of perspective taking and executive function to effective communication in nonaphasic human patients with behavioral variant frontotemporal dementia (bvFTD). Similar to previous work, the task was to identify a target object, embedded among an array of competitors, for an interlocutor. Results indicate that bvFTD patients are impaired relative to control subjects in selecting the optimal, precise response. Neuropsychological testing related this performance to mental set shifting, but not to working memory or inhibition. Follow-up analyses indicated that some bvFTD patients perform equally well as control subjects, while a second, clinically matched patient group performs significantly worse. Importantly, the neuropsychological profiles of these subgroups differed only in set shifting. Finally, structural MRI analyses related patient impairment to gray matter disease in orbitofrontal, medial prefrontal, and dorsolateral prefrontal cortex, all regions previously implicated in social cognition and overlapping those related to set shifting. Complementary white matter analyses implicated uncinate fasciculus, which carries projections between orbitofrontal and temporal cortices. Together, these findings demonstrate that impaired referential communication in bvFTD is cognitively related to set shifting, and anatomically related to a social-executive network including prefrontal cortices and uncinate fasciculus.