It is empirically known that fish exposed to harmful algal blooms (HABs) exhibit abnormal behavior. This might serve as a method for early detection of HABs. There has been no report of the detection of behavioral abnormalities of fish exposed to harmful algae using machine learning. In this study, the behavior of Oryzias javanicus (Java medaka) exposed in a stepwise manner to the HAB species Karenia mikimotoi at densities of 0 cells mL−1 (control), 1 × 103 cells mL−1 (nonlethal), and 5 × 103 cells mL−1 (sublethal) was recorded for 30 min at each cell density using two digital cameras connected to a software that tracked behavioral metrics of fish. The level of anomaly in the behavior of Java medaka was then analyzed using one‐class support vector machines (OC‐SVM) to determine whether the behavioral changes could be considered abnormal. The results revealed abnormal swimming behavior evidenced by an increase of swimming speed, a decrease of shoaling behavior, and a greater depth of swimming in Java medaka exposed especially to the sublethal K. mikimotoi density. The medaka exposed to K. mikimotoi also displayed physical deformities of their gills that were thought to have caused their abnormal behavior. This supposition was confirmed by further analysis using OC‐SVM because the behavior of groups exposed to nonlethal and sublethal densities of K. mikimotoi were considered abnormal compared with that of the control groups. The results of this study show the possibility of using this system for early and real‐time detection of HABs.