In visual perspective taking (vPT) one has to concern oneself with what other people see and how they see it. Since seeing is a mental state, developmental studies have discussed vPT within the domain of “theory of mind (ToM)” but imaging studies have not treated it as such. Based on earlier results from several meta-analyses, we tested for the overlap of visual perspective taking studies with 6 different kinds of ToM studies: false belief, trait judgments, strategic games, social animations, mind in the eyes, and rational actions. Joint activation was observed between the vPT task and some kinds of ToM tasks in regions involving the left temporoparietal junction (TPJ), anterior precuneus, left middle occipital gyrus/extrastriate body area (EBA), and the left inferior frontal and precentral gyrus. Importantly, no overlap activation was found for the vPT tasks with the joint core of all six kinds of ToM tasks. This raises the important question of what the common denominator of all tasks that fall under the label of “theory of mind” is supposed to be if visual perspective taking is not one of them.