Computer-aided diagnosis systems (CADs) can quantify the severity of diseases by analyzing a set of images and employing prior statistical models. In general, CADs have proven to be effective at providing quantitative measurements of the extent of a particular disease, thus helping physicians to better monitor the progression of cancer, infectious diseases, and other health conditions. Electronic Health Records frequently include a large amount of clinical data and medical history that can provide critical information about the underlying condition of a patient. We hypothesize that the fusion of image and clinical-physiological features can be used to enhance the accuracy of automatic image classification models. In particular, this paper shows how image analytic tools can move beyond classical image interpretation models to broader systems where image and physiological measurements are fused and used to create more generic detection models. To test our hypothesis, a CAD system capable of quantifying the severity of patients with pulmonary fibrosis has been developed. Results show that CAD systems augmented with multimodal physiological values are more robust and accurate at determining the severity of the disease.