Higher sedentary time and lower physical activity (PA) are associated with a poor health-related quality of life (HRQoL) in children. The aims of this study were: (1) to analyze the sedentary time, objectively measured PA levels (light, moderate, vigorous, and moderate-to-vigorous physical activity (MVPA)), and HRQoL dimensions (physical well-being, emotional well-being, self-esteem, family, friends, school, and total score) in children; and (2) to examine the association between sedentary time, PA levels, and HRQoL in children separately by sex. A total of 459 children (8.4 ± 0.4 years old, 50.54% males) from 15 schools in Granada (Spain) participated in the study. A tri-axial accelerometer was used to measure PA levels in the children for 7 consecutive days. The Revidierter KINDer Lebensqualitätsfragebogen (KINDL-R) questionnaire was used to determine the children’s HRQoL dimensions. The results showed that males presented more minutes engaged in MVPA than females. Both sedentary time and PA levels were associated with self-esteem and total score (all p < 0.05). In males, moderate and vigorous PA levels were associated with higher HRQoL, whereas light PA was associated with higher HRQoL in females. Future studies should take into account the use of activities with difference intensities in order to increase HRQoL in males and females.