An eye-tracking experiment was conducted to investigate the effects of incongruous contextual cues, both perceptual and conceptual, on pictorial metaphor processing. In a metaphoricity decision task, 38 participants independently viewed a total of 36 pictures equally divided into three different conditions: metaphoric pictures (MP), anomalous literal pictures (ALP) with perceptual congruity, and congruous literal pictures (LP). By initiating the midway condition ALP, the effect of contextual cues of conceptual incongruity were distinguished from that of perceptual incongruity. The eye movement data during each picture viewing were collected before the participant made a decision whether the picture was metaphorical or not. The behavioral results showed that the more abundant incongruous contextual cues were there, the more likely the pictures would be judged as metaphors. It took longer for the participants to make decisions on the literal pictures, be them in congruous or incongruous condition. The results of eye-tracking statistics showed that the perceptual incongruity cues were detected earlier than the conceptual ones. The perceptual-conceptual incongruity cues evoked more fixations and longer duration than the perceptual incongruity alone. The processing of conceptual incongruity proceeded after that of perceptual incongruity. The overall result of the study supports the contextual cues of perceptual incongruity as triggers for pictorial metaphor processing, whereas the contextual cues of conceptual incongruity play a decisive role in the metaphoric interpretation, which in turn renders the processing of MP more mental effort consuming than that of ALP or LP. The present findings have vital implications in revealing the triggering and determining mechanisms of pictorial metaphor processing, which are significant in exploring human cognition and have great impacts on various facets of social and cultural communications.