Fungal phytotoxic secondary metabolites are poisonous substances to plants produced by fungi through naturally occurring biochemical reactions. These metabolites exhibit a high level of diversity in their properties, such as structures, phytotoxic activities, and modes of toxicity. They are mainly isolated from phytopathogenic fungal species in the genera of Alternaria, Botrytis, Colletotrichum, Fusarium, Helminthosporium, and Phoma. Phytotoxins are either host specific or non-host specific phytotoxins. Up to now, at least 545 fungal phytotoxic secondary metabolites, including 207 polyketides, 46 phenols and phenolic acids, 135 terpenoids, 146 nitrogen-containing metabolites, and 11 others, have been reported. Among them, aromatic polyketides and sesquiterpenoids are the main phytotoxic compounds. This review summarizes their chemical structures, sources, and phytotoxic activities. We also discuss their phytotoxic mechanisms and structure–activity relationships to lay the foundation for the future development and application of these promising metabolites as herbicides.