Physical activity and cognitive functioning are strongly intertwined. However, the causal relationships underlying this association are still unclear. Physical activity can enhance brain functions, but healthy cognition may also promote engagement in physical activity. Here, we used Latent Heritable Confounder Mendelian Randomization (LHC-MR) to assess the bidirectional relations between physical activity and general cognitive functioning. Association data were drawn from two large-scale genome-wide association studies (UK Biobank and COGENT) on accelerometer-based physical activity (N = 91,084) and cognitive functioning (N = 257,841). We observed a significant MR association, suggesting that increased duration of physical activity improves cognitive functioning (b = 0.61, CI95% = [0.36,0.86], P = 1.16e-06). In contrast, we found no evidence for a causal effect of cognitive functioning on physical activity. Follow-up analyses revealed that the favorable association from physical activity to cognitive functioning was driven by moderate physical activity (b = 1.33, CI95% = [0.72,1.94], P = 2.01e-05) with no contribution from vigorous physical activity. These findings provide new evidence supporting a beneficial causal effect of moderate physical activity on cognitive functioning. Therefore, interventions that promote moderate rather than vigorous physical activity may be best suited to improve or recover cognitive skills.