The level of physical education (PE) is closely related to the cultivation of students' comprehensive quality and ability. Driven by big data, this paper deeply analyzes the importance, current situation and shortcomings of physical education evaluation system, and proposes an intelligent physical education evaluation model based on deep learning (DL) and data mining (DM) for these shortcomings. The model not only significantly improves the prediction accuracy and reduces the error, but more importantly, it deeply digs the multi-dimensional characteristics of students' sports ability through deep learning technology, and realizes a comprehensive and objective evaluation of students' sports level. The simulation results show that compared with the teaching evaluation model based on genetic algorithm, the prediction accuracy of this method is improved by 16.78% and the error is reduced by 22.66%. This method is obviously superior to the traditional method in classification performance. On the data set 1, the accuracy rate of the method reaches 96.8%, which is higher than that of the traditional methods (90.1% and 92.4%). 50% cross-validation shows that the average accuracy of this method is 96.2%, which is more stable. Therefore, it is feasible to apply this model to physical education evaluation, which provides theoretical support for the design and optimization of physical education evaluation index system driven by big data.