To facilitate pre-symptomatic diagnosis of late-onset Alzheimers disease, non-invasive imaging biomarkers could be combined with genetic risk information. In this work, we investigated the structural brain networks of young adults in relation to polygenic risk for Alzheimers disease, using magnetic resonance imaging (MRI) and genotype data for 564 19-year-old participants from the Avon Longitudinal Study of Parents and Children. Diffusion MRI was acquired on a 3T scanner, and the data were used to perform whole-brain tractography. The resulting tractograms were used to generate structural brain networks, using the number of streamlines and the diffusion tensor fractional anisotropy as edge weights. This was done for the whole-brain connectome, and for the default mode, limbic and visual subnetworks. Graph theoretical metrics were calculated for these networks, for each participant. The hubs of the networks were also identified, and the connectivity of the rich-club, feeder and local connections was also calculated. Polygenic risk scores (PRS), estimating the burden of genetic risk carried by an individual, were calculated both at genome-wide level and for nine specific disease pathways. The correlation coefficients were calculated between the PRSs and a) the graph theoretical metrics of the structural networks and b) the rich-club, feeder and local connectivity of the whole brain networks. In the visual subnetwork, the mean nodal strength exhibited a negative correlation with the genome-wide PRS including the APOE locus, while the mean betweenness centrality showed a positive correlation with the pathway-specific PRS for plasma lipoprotein particle assembly including the APOE locus. The rich-club connectivity was reduced in participants with higher genome-wide PRS including the APOE locus. Our results indicate small changes in the brain connectome of young adults at risk of developing Alzheimers disease in later life