As a compulsory course in universities, physical fitness is an important part of public health. At present, research on physical fitness lacks a comprehensive evaluation method, which cannot accurately guide public health. Based on this, this paper compares the key factors affecting public health, aiming to better guide public health and improve the level of public health. In this paper, 219 college students were selected as the research object, using statistical analysis methods, to find out the significant differences in influencing factors. Then, regression analysis is carried out on different influencing factors to determine the key factors affecting physical fitness. The results show that physical fitness level, exercise frequency, and physical injury are the key factors affecting physical fitness and have a serious impact on public health, with an impact level of about 70~80%. Therefore, colleges and sports departments should set out from the above factors, formulate a development plan for physical fitness, and promote the improvement of public health.