Prominent models of overt and covert visual search focus on explaining search efficiency by visual guidance. That some searches are fast whereas others are slow is explained by the ability of the target to guide attention to the target's position. Comparably little attention is given to other variables that might also influence search efficiency, such as dwelling on distractors, skipping distractors, and revisiting distractors. Here, we examine the relative contributions of dwelling, skipping, rescanning, and the use of visual guidance, in explaining visual search times in general, and the similarity effect in particular. The hallmark of the similarity effect is more efficient search for a target that is dissimilar to the distractors compared to a target that is similar to the distractors. In the present experiment, participants have to find an emotional face target among nine neutral face non-targets. In different blocks, the target is either more or less similar to the non-targets. Eye-tracking is used to separately measure selection latency, dwelling on distractors, and skipping and revisiting of distractors. As expected, visual search times show a large similarity effect. Similarity also has strong effects on dwelling, skipping, and revisiting, but only weak effects on visual guidance. Regression analyses show that dwelling, skipping, and revisiting determine search times on trial level. The influence of dwelling and revisiting is stronger in target absent than in target present trials, whereas the opposite is true for skipping. The similarity effect is best explained by dwelling. Additionally, including a measure of guidance does not yield substantial benefits. In sum, results indicate that guidance by the target is not the sole principle behind fast search; rather, distractors are less often skipped, more often visited, and longer dwelled on in slow search conditions.