Nowadays, fish farming plays an essential role in aquaculture. In the process of fry sales, the traditional manual counting method takes a great deal of time. Because the fish are constantly moving in the water, they are difficult to count and are likely to be unnecessarily damaged. With the rise of computer vision, fry counting based on image processing is also developing. It has the advantages of having high accuracy and fast processing speed. The primary counting process is divided into detection, adhesion segmentation, and fry counting. First, we improved the maximum interclass variance threshold selection method, and we extracted the histogram features under different channels to detect the fry; then, a matching segmentation method based on concave and skeleton crossing points was used to segment the adherent fry to improve the counting accuracy, the concaves were detected by CNN. We tested the method on real fry data sets, and the average accuracy reached 97%. Our approach had better accuracy than any existing current fish counting methods. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.