Fungal plant pathogens cause major crop losses worldwide, with many featuring compartmentalised genomes that include both core and accessory regions, which are believed to drive adaptation. The highly host‐specific fungus Colletotrichum lupini greatly impacts lupin (Lupinus spp.) cultivation. This pathogen is part of clade 1 of the C. acutatum species complex and comprises four genetically uniform, presumably clonal, lineages (I–IV). Despite this, variation in virulence and morphology has been observed within these lineages. To investigate the potential sources of genetic variability in this asexual fungus, we compared the genomes of 16 C. lupini strains and 17 related Colletotrichum species. Phylogenomics confirmed the presence of four distinct lineages, but further examination based on genome size, gene content, transposable elements (TEs), and deletions revealed that lineage II could be split into two groups, II‐A and II‐B. TE content varied between lineages and correlated strongly with genome size variation, supporting a role for TEs in genome expansion in this species. Pangenome analysis revealed a highly variable accessory genome, including a minichromosome present in lineages II, III, and IV, but absent in lineage I. Accessory genes and effectors appeared to cluster in proximity to TEs. Presence/absence variation of putative effectors was lineage‐specific, suggesting that these genes play a crucial role in determining host range. Notably, no effectors were found on the TE‐rich minichromosome. Our findings shed light on the potential mechanisms generating genetic diversity in this asexual fungal pathogen that could aid future disease management.