The inverse base-rate effect (IBRE) describes an apparent irrationality in human decision making whereby people tend to ignore category base rates and choose rarer options when classifying ambiguous stimuli. According to associative learning theories, people choose rare categories for ambiguous stimuli because rare cues draw more attention. Alternatively, inferential theories predict that people choose rare categories because ambiguous stimuli contrast more with well-established rules that predict membership in the common category. In this experiment, we used multi-voxel pattern analysis to decode which features of ambiguous stimuli, common or rare, participants are activating during an fMRI version of the IBRE task. Participants learned to predict four hypothetical diseases based on pairings of face, object, and scene cues. Objects and scenes were diagnostic cues, and predicted either a common or rare disease. Prior to the task, we collected independent localizer scans to characterize activation patterns associated with each of the different visual cues in object-selective cortex. During a test phase, ambiguous trials were presented in which a cue for the rare disease was paired with a cue for the common disease. We show that individuals engage qualitatively distinct neural processes when making rare versus common responses: choosing the rare category involved activation of cues associated with the common category. Consistent with inferential theories of base-rate neglect, our findings suggest that this surprising behavior involves a deliberative mechanism not explained by purely associative models.