Many studies have focused on navigation, spatial skills, and the olfactory system in comparative models, including those concerning the relationship between them and physical activity. Although the results are often in contrast with each other, it is assumed that physical activity can affect cognition in different ways—both indirectly and through a certain influence on some brain structures. In contrast, there is little research that focuses on the relationship between spatial abilities and olfactory abilities in humans. This research aimed to evaluate and compare the performance in working memory tasks of athletes and non-athletes who require good visual–spatial navigation, olfactory–spatial navigation, and olfactory–semantic skills. The study involved 236 participants (83 athletes) between the ages of 18 and 40. All subjects were matched by age or sex. The standard Corsi Block Tapping Test (CBTT) was administrated to investigate the visual-spatial memory. Olfactory–spatial navigation and olfactory–semantic skills were assessed with two modified versions of CBTT: Olfactory CBTT (OCBTT) and Semantic–Olfactory CBTT (SOCBTT) respectively. The results show differences between the CORSI conditions in direction of a poor performance for athletes. A gender effect in favor of men was also found, particularly in the classic version of the CBTT. Both groups performed better in the classic version of the CBTT than OCBTT and SOCBTT. The mean of SOCBTT results is markedly lower, perhaps due to the different information processing systems needed to perform this kind of task. It is possible to explain how sports practice can affect tasks that require spatial skills and olfactory perception differently, thus supporting new hypotheses and opening new scientific horizons.