The utilization of allelochemicals to inhibit algal overgrowth is a promising approach for controlling harmful algal blooms (HABs). Quercetin has been found to have an allelopathic effect on algae. However, its responsive mechanism needs to be better understood. In the present study, the inhibitory effects of different quercetin concentrations on M. aeruginosa were evaluated, and the inhibition mechanisms were explored. The results demonstrated that quercetin significantly inhibited M. aeruginosa growth, and the inhibitory effect was concentration-dependent. The inhibition rate of 40 mg L−1 quercetin on algal density reached 90.79% after 96 h treatment. The concentration of chlorophyll-a (chl-a) in treatment groups with quercetin concentrations of 10, 20, and 40 mg L−1 decreased by 59.74%, 74.77%, and 80.66% at 96 h, respectively. Furthermore, quercetin affects photosynthesis and damages the cell membrane, respiratory system, and enzyme system. All photosynthetic fluorescence parameters, including the maximum photochemical quantum yield (Fv/Fm), the actual photochemical quantum yield (YII), the maximum relative electron transfer rate (rETRmax), and light use efficiency (α), exhibited a downtrend after exposure. After treatment with 20 mg L−1 quercetin, the nucleic acid and protein content in the algal solution increased, and the respiration rate of algae decreased significantly. Additionally, superoxide dismutase (SOD) activities significantly increased as a response to oxidative stress. In comparison, the activities of ribulose 1,5-biphosphate carboxylase (Rubisco) and phosphoenolpyruvate carboxylase (PEPC) decreased significantly. These results revealed that quercetin could inhibit M. aeruginosa by affecting its photosynthesis, respiration, cell membrane, and enzymic system. These results are promising for controlling M. aeruginosa effectively.