Face exposure during development determines adults' abilities to recognize faces and the information they use to process them. Individual differences in the face categories represented in the visual environment can lead to category-specific deficits for recognizing faces that are atypical of observer's experience (e.g. the other-race effect). But what happens when observers have limited opportunities to learn about faces in general? In previous work, we found that observers from depopulated areas have poorer face recognition performance than observers from larger communities, suggesting that impoverished face experience limits face processing broadly. Here, we further investigate this phenomenon by examining how hometown size impacts the ability to assess appearance variability in natural images of faces and bodies. We asked individuals from small and large communities to complete (1) an unconstrained card-sorting task designed to test observers' ability to categorize within-person and between-person appearance variability properly, and (2) the Cambridge Face Memory Test. For both tasks, we examined the direct comparison between groups as well as the relationship between CFMT scores and sorting performance as a function of face experience. We find that small-town observers perform more poorly on the CFMT, but exhibit both better and worse performance than large-town observers on different aspects of the card-sorting task. Further, we also examine the relationship between CFMT performance and card-sorting errors. Our results suggest that individual differences in lifetime face exposure induce important variation in face processing abilities.