Previous studies have confirmed that attention can be modulated by the current task set while involuntarily captured by salient items. However, little is known on which factors the modulation of attentional capture is dependent on when the same stimuli with different task sets are presented. In the present study, participants conducted two visual search tasks with the same search arrays by varying target and distractor settings (color singleton as target, onset singleton as distractor, named as color task, and vice versa). Ipsilateral and contralateral color distractors resulted in two different relative saliences in two tasks, respectively. Both reaction times (RTs) and N2-posterior-contralateral (N2pc) results showed that there was no difference between ipsilateral and contralateral color distractors in the onset task. However, both RTs and the latency of N2pc showed a delay to the ipsilateral onset distractor compared with the contralateral onset distractor. Moreover, the N2pc observed under the contralateral distractor condition in the color task was reversed, and its amplitude was attenuated. On the basis of these results, we proposed a parameter called distractor cost (DC), computed by subtracting RTs under the contralateral distractor condition from the ipsilateral condition. The results suggest that an enhanced DC might be related to the modification of N2pc in searching for the color target. Taken together, these findings provide evidence that the effect of task set-modulating attentional capture in visual search is related to the DC.