Endophytic fungi strains (n = 81) were isolated from the leaves, barks, and fruits of Camellia oleifera from Hunan province (China) to delineate their species composition and potential as biological control agents of C. oleifera anthracnose. The fungi were identified by morphological and phylogenetic analyses. Fungal colonization rates of the leaves, barks, and fruits were 58.02, 27.16, and 14.81%, respectively. The isolates were identified as 14 genera, belonging to two subdivisions, Deuteromycotina and Ascomycotina; 87.65% of all isolates belonged to Deuteromycotina. The dominant species, occurring with a high relative frequency, were Pestalotiopsis sp. (14.81%), Penicillium sp. (14.81%), and Fusarium sp. (12.35%). The Simpson’s and Shannon’s diversity indices revealed the highest species diversity in the leaves, followed by the barks and fruits. The similarity index for the leaves versus barks comparison was the highest, indicating that the number of endophytic fungal species shared by the leaves and barks was higher than barks and fruits or leaves and fruits. Based on the results of dual culture experiments, only five strains exhibited antifungal activity against C. oleifera anthracnose pathogen, with isolate ty-64 (Oidium sp.) generating the broadest inhibition zones. Our results indicate that the endophytes associated with C. oleifera could be employed as natural agents controlling C. oleifera anthracnose.