Being born at extremely low birth weight (ELBW; ≤1,000 g) is associated with enduring visual impairments. We tested for long‐term, higher order visual processing problems in the oldest known prospectively followed cohort of ELBW survivors. Configural processing (spacing among features of an object) was examined in 62 adults born at ELBW (Mage = 31.9 years) and 82 adults born at normal birth weight (NBW; ≥2,500 g: Mage = 32.5 years). Pairs of human faces, monkey faces, or houses were presented in a delayed match‐to‐sample task, where non‐matching stimuli differed only in the spacing of their features. Discrimination accuracy for each stimulus type was compared between birth weight groups, adjusting for neurosensory impairment, visual acuity, binocular fusion ability, IQ, and sex. Both groups were better able to discriminate human faces than monkey faces (p < .001). However, the ELBW group discriminated between human faces (p < .001), between monkey faces (p < .001), and to some degree, between houses (p < .06), more poorly than NBW control participants, suggesting a general deficit in perceptual processing. Human face discrimination was related to performance IQ (PIQ) across groups, but especially among ELBW survivors. Coding (a PIQ subtest) also predicted human face discrimination in ELBW survivors, consistent with previously reported links between visuo‐perceptive difficulties and regional slowing of cortical activity in individuals born preterm. Correlations with Coding suggested ELBW survivors may have used a feature‐matching approach to processing human faces. Future studies could examine brain‐based anatomical and functional evidence for altered face processing, as well as the social and memory consequences of face‐processing deficits in ELBW survivors.