An object’s location can be represented either relative to an observer’s body effectors (egocentric reference frame) or relative to another external object (allocentric reference frame). In non-spatial tasks, an object’s task-irrelevant egocentric position conflicts with the side of a task-relevant manual response, which defines the classical Simon effect. Growing evidence suggests that the Simon effect occurs not only based on conflicting positions within the egocentric but also within the allocentric reference frame. Although neural mechanisms underlying the egocentric Simon effect have been extensively researched, neural mechanisms underlying the allocentric Simon effect and their potential interaction with those underlying its egocentric variant remain to be explored. In this fMRI study, spatial congruency between the task-irrelevant egocentric and allocentric target positions and the task-relevant response hand was orthogonally manipulated. Behaviorally, a significant Simon effect was observed for both reference frames. Neurally, three sub-regions in the frontoparietal network were involved in different aspects of the Simon effect, depending on the source of the task-irrelevant object locations. The right precentral gyrus, extending to the right SMA, was generally activated by Simon conflicts, irrespective of the spatial reference frame involved, and showed no additive activity to Simon conflicts. In contrast, the right postcentral gyrus was specifically involved in Simon conflicts induced by task-irrelevant allocentric, rather than egocentric, representations. Furthermore, a right lateral frontoparietal network showed increased neural activity whenever the egocentric and allocentric target locations were incongruent, indicating its functional role as a mismatch detector that monitors the discrepancy concerning allocentric and egocentric object locations.