Cyclic pentapeptide compounds have garnered much attention as a drug discovery resource. This study focused on the characterization and anti-benign prostatic hyperplasia (BPH) properties of avellanin A from Aspergillus fumigatus fungus in marine sediment samples collected in the Beibu Gulf of Guangxi Province in China. The antiproliferative effect and molecular mechanism of avellanin A were explored in testosterone propionate (TP)-induced RWPE-1 cells. The transcriptome results showed that avellanin A significantly blocked the ECM–receptor interaction and suppressed the downstream PI3K-Akt signalling pathway. Molecular docking revealed that avellanin A has a good affinity for the cathepsin L protein, which is involved in the terminal degradation of extracellular matrix components. Subsequently, qRT-PCR analysis revealed that the expression of the genes COL1A1, COL1A2, COL5A2, COL6A3, MMP2, MMP9, ITGA2, and ITGB3 was significantly downregulated after avellanin A intervention. The Western blot results also confirmed that it not only reduced ITGB3 and FAK/p-FAK protein expression but also inhibited PI3K/p-PI3K and Akt/p-Akt protein expression in the PI3K-Akt signalling pathway. Furthermore, avellanin A downregulated Cyclin D1 protein expression and upregulated Bax, p21WAF1/Cip1, and p53 proapoptotic protein expression in TP-induced RWPE-1 cells, leading to cell cycle arrest and inhibition of cell proliferation. The results of this study support the use of avellanin A as a potential new drug for the treatment of BPH.