Visual decisions often involve comparisons of sequential stimuli that can appear at any location in the visual field. The lateral prefrontal cortex (LPFC) in nonhuman primates, shown to play an important role in such comparisons, receives information about contralateral stimuli directly from sensory neurons in the same hemisphere, and about ipsilateral stimuli indirectly from neurons in the opposite hemisphere. This asymmetry of sensory inputs into the LPFC poses the question of whether and how its neurons incorporate sensory information arriving from the two hemispheres during memory-guided comparisons of visual motion. We found that, although responses of individual LPFC neurons to contralateral stimuli were stronger and emerged 40 ms earlier, they carried remarkably similar signals about motion direction in the two hemifields, with comparable direction selectivity and similar direction preferences. This similarity was also apparent around the time of the comparison between the current and remembered stimulus because both ipsilateral and contralateral responses showed similar signals reflecting the remembered direction. However, despite availability in the LPFC of motion information from across the visual field, these "comparison effects" required for the comparison stimuli to appear at the same retinal location. This strict dependence on spatial overlap of the comparison stimuli suggests participation of neurons with localized receptive fields in the comparison process. These results suggest that while LPFC incorporates many key aspects of the information arriving from sensory neurons residing in opposite hemispheres, it continues relying on the interactions with these neurons at the time of generating signals leading to successful perceptual decisions.