In table tennis, the ball has numerous characteristics of high speed, small size, and changeable trajectory. Due to these characteristics, the human eye often cannot accurately judge the ball’s movement and position, leading to the problem of precise detection of the ball’s falling point and movement tracking. In sports, the use of machine learning for locating and detecting the ball and the use of deep learning for reconstructing and displaying the ball’s trajectories are considered futuristic technologies. Therefore, this paper proposes a novel algorithm for identifying and scoring points in table tennis based on dual-channel target motion detection. The proposed algorithm consists of multiple input channels to jointly learn different features of table tennis images. The original image is used as the input of the first channel, and then the Sobel operator is used to extract the first-order derivative feature of the original image, which is used as the input of the second channel. The table tennis feature information from the two channels is then fused and sent to the 3D neural network module. The fully connected layer is used to identify the table tennis ball’s drop point, compare it with a standard drop point, calculate the error distance, and give a score. We also constructed a data set and conducted experiments. The experimental results show that the method in this paper is effective in sports.