A striking relationship between visual spatial perspective taking (VSPT) and social skills has been demonstrated for perspective-taking tasks in which the target of the imagined or inferred perspective is a potential agent, suggesting that the presence of a potential agent may create a social context for the seemingly spatial task of imagining a novel visual perspective. In a series of studies, we set out to investigate how and when a target might be viewed as sufficiently agent-like to incur a social influence on VSPT performance. By varying the perceptual and conceptual features that defined the targets as potential agents, we find that even something as simple as suggesting animacy for a simple wooden block may be sufficient. More critically, we found that experience with one potential agent influenced the performance with subsequent targets, either by inducing or eliminating the influence of social skills on VSPT performance. These carryover effects suggest that the relationship between social skills and VSPT performance is mediated by a complex relationship that includes the task, the target, and the context in which that target is perceived. These findings highlight potential problems that arise when identifying a task as belonging exclusively to a single cognitive domain and stress instead the highly interactive nature of cognitive domains and their susceptibility to cross-domain individual differences.