Five experiments were conducted to examine the performance of young infants on above versus below categorization tasks. Experiments 1 and 2 showed that infants did not form abstract categorical representations for above and below when familiarized with different objects depicted in a constant spatial relation relative to a horizontal bar and tested on a novel object depicted in the familiar and novel spatial relation. Experiments 3 through 5 examined perceptual‐attentional distraction versus conceptually based generalization explanations for young infant performance in the object‐variation version of the above‐below categorization task. Experiments 3 and 4 demonstrated that infants still did not form abstract categorical representations for above and below when object variation was removed from the familiarization trials or when object novelty was reduced during the preference test trials. However, Experiment 5 showed that 3‐ and 4‐month‐olds succeeded on the above versus below categorization task when familiarized with object variation and preference tested with a familiar versus novel object‐bar relation. These results indicate that young infants can form categorical representations for above and below in the object‐variation version of the above‐below categorization task, but that such representations are specific to the particular objects presented. Young infant performance in the object‐variation version of the above‐below categorization task thus reflects a conceptually based generalization limit rather than a problem of perceptual‐attentional distraction.