With the development of economy and culture and the improvement of infrastructure, basketball is becoming more and more popular. At the same time, whether in school or society, basketball events are increasing, and the demand for basketball referees is also rising. As the base and cradle of training basketball referees, colleges and universities are important places for incubating basketball referees. However, the basketball referee ability cultivated in colleges and universities is still lacking, which cannot meet the increasingly standardized basketball events and basketball training. Infrared thermal imaging has the characteristics of passive imaging, high recognition accuracy, and strong anti-interference. It is especially suitable for tracking and testing basketball players, and this is expected to further improve the comprehensive ability of basketball referees. Based on this, a target recognition and tracking system based on infrared thermal imaging is proposed in this study. Its purpose is to use it to carry out comprehensive ability training for college basketball referees. Experiments show that target recognition and tracking based on infrared thermal imaging can improve the comprehensive ability of basketball referees by 40.32%, and the awareness of basketball referees to fouls has also reached 88.6%. This fully shows that the target recognition and tracking system based on infrared thermal imaging can effectively improve the comprehensive ability of basketball referees and make basketball more regular and standardized.