Childhood attention deficit hyperactivity disorder (ADHD) shows a highly variable course with age: some individuals show improving, others stable or worsening symptoms. The ability to predict symptom course could help individualize treatment and guide interventions. By studying a cohort of 362 youth, we ask if polygenic risk for ADHD, combined with baseline neural and cognitive features could aid in the prediction of the course of symptoms over an average period of 4.8 years. Compared to a never affected comparison group, we find that participants with worsening symptoms carried the highest polygenic risk for ADHD, followed by those with stable symptoms, then those whose symptoms improved. Participants with worsening symptoms also showed atypical baseline cognition. Atypical microstructure of the cingulum bundle and anterior thalamic radiation were associated with improving symptoms while reduction of thalamic volume was found in those with stable symptoms. Machine learning algorithms, trained and tested on independent groups, performed well in classifying those never affected against groups with worsening, stable and improving symptoms (area under the curve > 0.79). We conclude that some measures of polygenic risk, cognition, and neuroimaging show significant associations with the future course of ADHD symptoms and may have modest predictive power. These features warrant further exploration as prognostic tools.