In boxing, as the referee makes decisions through punting, a referee evaluation system allows for adequate analysis of the vast amount of punting data. The aim of this paper is to study a comprehensive statistical assessment system for boxing referees based on convolutional neural networks. The concept of comprehensive statistical assessment of referees is proposed and a data warehouse for comprehensive statistical assessment of referees is established. A multi-dimensional data query analysis is realised. Development of a video retrieval module for boxing systems using video retrieval technology. LeNet-5 convolutional neural network-based recognition of athlete's punching situation is proposed. The different effects of the learning rate on the recognition effect are then investigated, and it is experimentally demonstrated that the best results are obtained at a learning rate of 0.001.