(1) Background: COVID-19 continues to represent a worrying pandemic. Despite the high percentage of non-severe illness, a wide clinical variability is often reported in real-world practice. Accurate predictors of disease aggressiveness, however, are still lacking. The purpose of our study was to evaluate the impact of quantitative analysis of lung computed tomography (CT) on non-intensive care unit (ICU) COVID-19 patients’ prognostication; (2) Methods: Our historical prospective study included fifty-five COVID-19 patients consecutively submitted to unenhanced lung CT. Primary outcomes were recorded during hospitalization, including composite ICU admission for the need of mechanical ventilation and/or death occurrence. CT examinations were retrospectively evaluated to automatically calculate differently aerated lung tissues (i.e., overinflated, well-aerated, poorly aerated, and non-aerated tissue). Scores based on the percentage of lung weight and volume were also calculated; (3) Results: Patients who reported disease progression showed lower total lung volume. Inflammatory indices correlated with indices of respiratory failure and high-density areas. Moreover, non-aerated and poorly aerated lung tissue resulted significantly higher in patients with disease progression. Notably, non-aerated lung tissue was independently associated with disease progression (HR: 1.02; p-value: 0.046). When different predictive models including clinical, laboratoristic, and CT findings were analyzed, the best predictive validity was reached by the model that included non-aerated tissue (C-index: 0.97; p-value: 0.0001); (4) Conclusions: Quantitative lung CT offers wide advantages in COVID-19 disease stratification. Non-aerated lung tissue is more likely to occur with severe inflammation status, turning out to be a strong predictor for disease aggressiveness; therefore, it should be included in the predictive model of COVID-19 patients.