Common and rare genetic variants that impact adult cognitive performance also predispose to rare neurodevelopmental conditions involving cognitive deficits in children. However, their influence on cognition across early life remains poorly understood. Here, we investigate the contribution of common genome-wide and rare exonic variation to cognitive performance across childhood and adolescence primarily using the Avon Longitudinal Study of Parents and Children (n=6,495 unrelated children). We show that the effect of common variants associated with educational attainment and adult cognitive performance increases as children age. Conversely, the negative effect of deleterious rare variants attenuates with age. Using trio analyses, we show that these age-related trends are driven by direct genetic effects on the individual who carries these variants rather than indirect genetic effects mediated via the family environment. We further find that the increasing effects of common variants are stronger in individuals at the upper end of the phenotype distribution, whereas the attenuating effects of rare variants are stronger in those at the lower end. Concordant results were observed in the Millenium Cohort Study (5,920 children) and UK Biobank (101,232 adults). The effects of common and rare genetic variation on childhood cognitive performance are broadly comparable in magnitude to those of other factors such as parental educational attainment, maternal illness and preterm birth. The effects of maternal illness and preterm birth on childhood cognitive performance also attenuate with age, whereas the effect of parental educational attainment does not. We show that the relative contribution of these various factors differs depending on whether one considers their contribution to phenotypic variance across the entire population or to the risk of poor outcomes. Our findings may help explain the apparent incomplete penetrance of rare damaging variants associated with neurodevelopmental conditions. More generally, they also show the importance of studying dynamic genetic influences across the life course and their differential effects across the phenotype distribution.