The aim of this study was to determine the relationship between different physical fitness indices (PFIs) and psychological symptoms and each dimension (emotional symptoms, behavioral symptoms, social adaptation difficulties) of Chinese children and adolescents. Methods: A total of 7199 children and adolescents aged 13–18 in Jiangxi Province, China, were tested for grip strength, standing long jump, sit-ups, sit and reach, repeated straddling, 50 m run, 20 m shuttle run test (20 m SRT) items. The physical fitness indicators were standardized, converted to Z score and added up to obtain the PFI, and the self-assessment of the psychological section of the multidimensional sub-health questionnaire of adolescents (MSQA) to test the psychological symptoms, using the chi-square test to determine the psychological symptoms of different types of children and adolescents and binary logistic regression analysis to determine the association between psychological symptoms and different PFI grades. Results: The higher the PFI of Chinese children and adolescents, the lower the detection rate of psychological symptoms, emotional symptoms and social adaptation difficulties, from 25.0% to 18.4%, 31.3% to 25.7% and 20.1% to 14.4%, respectively. These results were statistically significant (χ2 = 14.073, 9.332, 12.183, p < 0.05). Taking the high-grade PFI as a reference, binary logistic regression analysis was performed. Generally, compared with the high-grade PFI, children and adolescents with a low-grade PFI (OR = 1.476, 95% CI: 1.200–1.814) or medium-grade PFI (OR = 1.195, 95% CI: 1.010–1.413) had a higher risk of psychological symptoms (p < 0.05). Conclusions: The lower the PFI of Chinese children and adolescents, the higher the detection rate of psychological symptoms, showing a negative correlation. In the future, measures should be taken to improve the physical fitness level of children and adolescents in order to reduce the incidence of psychological symptoms and promote the healthy development of children and adolescents.