Many frameworks have assessed the ultimate and ontogenetic underpinnings in the development of object permanence, but less is known about whether individual characteristics, such as sex or training level, as well as proximate factors, such as arousal or emotional state, affect performance in these tasks. The current study investigated horses’ performance in visible and invisible displacement tasks and assessed whether specific ontogenetic, behavioral, and physiological factors were associated with performance. The study included 39 Icelandic horses aged 2–25 years, of varying training levels. The horses were exposed to three tasks: (a) a choice test (n = 37), (b) a visible displacement task (n = 35), and (c) an invisible displacement task (n = 31). 27 horses in the choice test, and 8 horses in the visible displacement task, performed significantly better than expected by chance, while none did so in the invisible displacement task. This was also reflected in their group performance, where horses performed above chance level in the choice task and the visible displacement task only. In the invisible displacement task, the group performed significantly worse than expected by chance indicating that horses persistently chose the side where they had last seen the target. None of the individual characteristics included in the study had an effect on performance. Unsuccessful horses had higher heart rate levels, and expressed more behavior indicative of frustration, likely because of their inability to solve the task. The increased frustration/arousal could lead to a negative feedback loop, which might hamper performance in subsequent trials. Care should thus be taken in future experimental designs to closely monitor the arousal level of the tested individuals in order to safeguard comparability.