The method based on the photosynthetic inhibition effect of algae offers the advantages of swift response and straightforward measurement. Nonetheless, this effect is influenced by both the environment and the state of the algae themselves. Additionally, a single parameter is vulnerable to uncertainties, rendering the measurement accuracy and stability inadequate. This paper employed currently utilized photosynthetic fluorescence parameters, including Fv/Fm(maximum photochemical quantum yield), Performance Indicator (PIabs), Comprehensive Parameter Index (CPI) and Performance Index of Comprehensive Toxicity Effect (PIcte), as quantitative toxicity characteristic parameters. The paper compared the univariate curve fitting results with the multivariate data-driven model results and investigated the effectiveness of Back Propagation(BP) Neural Network and Support Vector Machine for Regression (SVR) models to enhance the accuracy and stability of toxicity detection. Using Dichlorophenyl Dimethylurea (DCMU) samples as an example, the mean Relative Root Mean Square Error (RRMSE) corresponding to the optimal parameter PIcte for the dose-effect curve fitting was 1.246 in the concentration range of 1.25–200 µg/L. On the other hand, the mean RRMSEs corresponding to the results of the BP neural network and SVR models were 0.506 and 0.474, respectively. Notably, BP neural network exhibited excellent prediction accuracy in the medium-high concentration range of 7.5–200 µg/L, with a mean RRSME of only 0.056. Regarding the stability of the results, the mean Relative Standard Deviation (RSD) of the univariate dose-effect curve results was 15.1% within the concentration range of 50–200 µg/L. In contrast, the mean RSDs for both BP neural network and SVR results were less than 5%. In the concentration range of 1.25–200 µg/L, the mean RSDs were 6.1% and 16.5%, with the BP neural network performing well. The experimental results of Atrazine were analyzed to further validate the effectiveness of the BP neural network in improving the accuracy and stability of results. These findings provided valuable insights for the development of biotoxicity detection by using the algae photosynthetic inhibition method.