We investigated whether distractor inhibition occurs relative to the target or fixation in a perceptual decision-making task using a purely saccadic response. During the process of discriminating a target from distractor, saccades made to a target deviate towards the distractor. Once discriminated, the distractor is inhibited, and trajectories deviate away from the distractor. Saccade deviation magnitudes provide a sensitive measure of target-distractor competition dependent on the distance between them. While saccades are planned in an egocentric reference frame (locations represented relative to fixation), object-based inhibition has been shown to occur in an allocentric reference frame (objects represented relative to each other independent of fixation). By varying the egocentric and allocentric distances of the target and distractor, we found that only egocentric distances contributed to saccade trajectories shifts towards the distractor during active decision-making. When the perceptual decision-making process was complete, and the distractor was inhibited, both ego- and allocentric distances independently contributed to saccade trajectory shifts away from the distractor. This is consistent with independent spatial and object-based inhibitory mechanisms. Therefore, we suggest that distractor inhibition is maintained in cortical visual areas with allocentric maps which then feeds into oculomotor areas for saccade planning.