Regular physical activity (PA) is indicated to be one of the main healthy habits that allow populations to achieve a good quality of life (QoL); however, levels of physical inactivity (PI) and sedentary behavior (SB) have risen worldwide, with negative health effects. The aim of this study is to analyze PI and SB levels, as well as their association with QoL in the Chilean population. A secondary analysis was performed based on the database from the 2015–2016 National Quality of Life Survey (ENCAVI) in Chile, using the modules for sociodemographic characteristics, health, and PA. Contingency tables and logistic regressions were conducted to determine the association between PI (low-intensity PA, LIPA; low–moderate PA, LMPA), SB (P75), and QoL adjusted for sociodemographic variables. Approximately 84%, 83%, and 47% of the participants presented LIPA, LMPA, and moderate–high SB, respectively. Participants that presented high PI and SB had lower QoL scores than those who were more active (p < 0.05). We observe that, in people with PI (LIPA), there is a higher risk of a low QoL regarding mental and physical components (OR 1.941; OR 1.189, p < 0.001) among females (OR 1.473; OR 1.513, p < 0.001) and those of a low educational level (OR 2.170; OR 1.410 p < 0.001). People with PI (LMPA) increased their risk for a low QoL in regard to mental and physical components (OR 1.750; OR 1.458, p < 0.001) among females (OR 1.528; OR 1.507, p < 0.001) and those of a low educational level (OR 2.195; OR 1.402 p < 0.001). We observe that people with SB (P75) increased their risk of a low QoL concerning physical and mental components (OR 1.475; OR 1.257, p < 0.001) for those of the female gender (OR 1.615; OR 1.563, p < 0.001) and a low educational level (OR 2.248; OR 1.423 p < 0.001). High levels of PI in both intensities and SB impact QoL in the Chilean population. It is crucial to generate public policies for more PA, especially for females and those of lower educational levels.