Population structure is an important factor that affects the accuracy of estimated breeding values in genomic prediction. Natural sorghum [Sorghum bicolor (L.) Moench] populations exhibit population structure resulting from genetic and morphological differentiation due to evolutionary divergence. To study the impact of sorghum racial structure and diversity in genomic prediction, we conducted two cross‐validation (CV) experiments: CV1, proportional sampling from races; and CV2, sampling from across race (AR) or within race (WR). A diversity panel with 389 individuals with 224,007 single nucleotide polymorphisms was used for genomic prediction. Genomic heritabilities for traits were positively correlated (0.63) with their mean prediction accuracy (r) from CV1, and within‐subpopulation variance accounted for ∼80% of total genetic variance. The CV1 prediction accuracy ranged from 0.52–0.69, but r declined by 39 and 54% on average for WR and AR methods, respectively. As a predictor, race explained 30–50% of covariance for grain and panicle traits, but race was a bad predictor of plant height, as expected. Grain weight was consistently the best predicted trait across CV1 and CV2 methods except in AR. Difference in average r for WR and AR was greater in durra and caudatum, small in kafir, and nonexistent in guinea and mixed subgroups. We observed higher prevalence of minor alleles among guinea and mixed subgroups, highlighting contribution of allelic diversity towards prediction accuracy. Genomic prediction in sorghum will benefit from utilization of interracial diversity, and we emphasize the need for further investigations into the role of racial structure in genomic prediction.