Toxicology studies heavily rely on morphometric analysis to detect abnormalities and diagnose disease processes. The emergence of ever-increasing varieties of environmental pollutants makes it difficult to perform timely assessments, especially using in vivo models. Herein, we propose a deep learning-based morphometric analysis (DLMA) to quantitatively identify eight abnormal phenotypes (head hemorrhage, jaw malformation, uninflated swim bladder, pericardial edema, yolk edema, bent spine, dead, unhatched) and eight vital organ features (eye, head, jaw, heart, yolk, swim bladder, body length, and curvature) of zebrafish larvae. A data set composed of 2532 bright-field micrographs of zebrafish larvae at 120 h post fertilization was generated from toxicity screening of three categories of chemicals, i.e., endocrine disruptors (perfluorooctanesulfonate and bisphenol A), heavy metals (CdCl 2 and PbI 2 ), and emerging organic pollutants (acetaminophen, 2,7-dibromocarbazole, 3-monobromocarbazo, 3,6-dibromocarbazole, and 1,3,6,8-tetrabromocarbazo). Two typical deep learning models, one-stage and two-stage models (TensorMask, Mask R-CNN), were trained to implement phenotypic feature classification and segmentation. The accuracy was statistically validated with a mean average precision >0.93 in unlabeled data sets and a mean accuracy >0.86 in previously published data sets. Such a method effectively enables subjective morphometric analysis of zebrafish larvae to achieve efficient hazard identification of both chemicals and environmental pollutants.