In current college music education, choral conducting is a required course for students. The course implementation aims to cultivate excellent and high-quality choral conductors. The requirements for choral conducting teaching in college music education under the new media environment have been further improved. First, this study gives the value of applying new media technology in choral conducting teaching in colleges and universities. Then, based on the key point that choral conductors’ expression of music mainly relies on gestural language, an action recognition model in college choral conducting teaching is proposed. The model is designed with an adaptive deep graph convolution model, and a spatio-temporal convolution submodel with a small number of parameters is created using group convolution. After the trained teacher model is obtained, the spatio-temporal convolutional submodel with fewer parameters is trained using the knowledge distillation method combined with data augmentation techniques. The final action recognition fusion model is obtained using the linear fusion method. The experimental results demonstrate that the proposed model can recognize the movements in college choral conducting teaching with higher performance than other existing models, which provides effective guidance for college choral conducting teaching in the new media environment.