The physical fitness of college students can be evaluated scientifically based on the data of physical education (PE). This paper firstly relies on the Apriori algorithm to mine the hidden correlations between the physical fitness indices from the PE data on college students, and identify the indices closely associated with the physical fitness of college students. Then, the Apriori algorithm was improved to reduce the time complexity of association rule mining. Based on the improved algorithm, it was learned that the correlation coefficients of several indices surpassed the minimum support of 0.2 and minimum confidence of 0.7, reflecting their important impacts on physical fitness. Thus, physical fitness of college students is significantly influenced by speed, endurance, flexibility, and vital capacity, but not greatly affected by height and weight. The research results provide an important guide for the test and curriculum designs of PE for college students.