Fungal diseases affect millions of humans annually, yet fungal pathogens remain understudied. The mold Aspergillus flavus is a causative agent of both aspergillosis and fungal keratitis infections, but species closely related to A. flavus are not considered clinically relevant. To study the evolution of A. flavus pathogenicity, we examined genomic and phenotypic traits of two strains of A. flavus and three closely related non-pathogenic species: Aspergillus arachidicola (two strains), Aspergillus parasiticus (two strains), and Aspergillus nomiae (one strain). We identified over 3,000 orthologous proteins unique to A. flavus, including seven biosynthetic gene clusters present in A. flavus strains and absent in the three non-pathogenic species. We chose to characterize secondary metabolite production for all seven strains under two clinically relevant conditions, temperature and salt concentration. Temperature impacted metabolite production in all species. Conversely, we found a lack of impact of salinity on secondary metabolite production. Strains of the same species produced different metabolites. Growth under stress conditions revealed additional heterogeneity within species. Using the invertebrate model of fungal disease Galleria mellonella, we found virulence of strains of the same species varied widely, and A. flavus strains were not more virulent than strains of the non-pathogenic species. In a murine model of fungal keratitis, we observed significantly lower disease severity and corneal thickness for A. arachidicola compared to other species at 48 hrs, but not at 72 hrs. Our work identifies key phenotypic, chemical, and genomic similarities and differences between the opportunistic human pathogen A. flavus and its non-pathogenic relatives.