In an environment busy with abundant stimulation, individuals must rely on attentional cues to select the most relevant targets. As social creatures, a crucial strategy involves monitoring where others are focusing their attention and following them, assuming that locations attended by others are more likely to yield benefits than other locations. Given that gaze shifts represent a visible shift in attention, mirroring the gaze shifts of others can serve as an effective, social-attentional strategy. Indeed, research demonstrates that observing others redirect their gaze results in a reflexive shift of attention, reflected by improved perceptual performance for objects located at the gazed-at positions. This phenomenon is known as thegaze cueing effect(GCE). However, gaze shifts may not always align with the direction of attention. Individuals often avert their gaze while they are engaged in demanding cognitive tasks. Such gaze shifts signify internal focus rather than shifting attention outward. Here we hypothesize that the response to a gaze-shift cue is not reflexive but is contingent upon social context. In two pre-registered experiments, participants viewed videos of gaze shifts and engaged in a perceptual task, with one group primed to perceive the gaze shifts as reflecting cognitive processing rather than overt attentional shifts. Results indicated that GCE was suppressed in this group, suggesting that GCE is influenced by contextual cues framing the nature of the gaze shifts. These findings suggest that social attention is flexible and depends on the specific interpretation of the social context.