There are a large number of 6.97 million left-behind children in China at the end of August 2018. Left-behind children exhibit many psychologically unhealthy behavioural disorders. This study aimed to compare the differences in scientific fitness literacy (SFL) between rural left-behind and general minors, and to analyze the factors affecting the SFL of left-behind minors in the family environment. A random stratified sampling was conducted among minors aged 3–18 in rural areas of China, and 2239 valid samples were obtained. The questionnaire is based on the SFL part and the family environment part of the China National Fitness Activity Status Survey, except for the SFL part of the children’s questionnaire does not contain the attitude dimension, the questionnaires of several other age groups contain three dimensions: attitude, skills, and habits. The SFL of general children and adolescents was higher than that of left-behind children and adolescents, and in the sub-dimensions, children’s attitudes and adolescents’ skills and habits differed greatly, with the general minors having better performance. The SFL showed a significant increasing trend with age in the general minors but the difference between children and adolescents in the left-behind minors was not significant. The scores of habituation in left-behind minors increased with age group, and the scores of the general minors group did not vary much between age groups. In the multiple regression analysis for the left-behind minors, it was found that left-behind young children were less influenced by family environment on SFL score, and children’s SFL was more influenced by family environment. A separate analysis of the 7–9 and 10–12-year-old groups revealed that factors such as parental support and encouragement influenced SFL with different weights, and the importance of parents rotated, while the number of sporting goods in the family was important in all four age models. For the left-behind minors with a large amount in China, the role of the family environment in their SFL cannot be ignored, and the promotion of this literacy should be carried out in a targeted manner.