Previous research suggests that closing one’s eyes or averting one’s gaze from another person can benefit visual-spatial imagination by interrupting cognitive demands associated with face-to-face interaction (Markson and Paterson, 2009). The present study further investigated this influence of social gaze on adults’ visual-spatial imagination, using the matrix task (Kerr, 1987, 1993). Participants mentally kept track of a pathway through an imaginary 2-dimensional (2D) or 3-dimensional (3D) matrix. Concurrent with this task, participants either kept their eyes closed or maintained eye contact with another person, mutual gaze with a person whose eyes were obscured (by wearing dark glasses), or unreciprocated gaze toward the face of a person whose own gaze was averted or whose face was occluded (by placing a paper bag over her head). Performance on the 2D task was poorest in the eye contact condition, and did not differ between the other gaze conditions, which produced ceiling performance. However, the more difficult 3D task revealed clear effects of social gaze. Performance on the 3D task was poorest for eye contact, better for mutual gaze, and equally better still for the unreciprocated gaze and eye-closure conditions. The findings reveal the especially disruptive influence of eye contact on concurrent visual-spatial imagination and a benefit for cognitively demanding tasks of disengaging eye contact during face-to-face interaction.