Background and Objectives: Both chronic pancreatitis (CP) and pancreatic ductal adenocarcinoma (PDAC) may lead to cachexia, sarcopenia, and osteoporosis due to different mechanisms. Neither patient gender, age, nor body weight are good predictors of these metabolic changes having a significant negative impact on the quality of life (QOL) and treatment outcomes. The aim of this study was to evaluate radiological changes in body composition and to compare them with manifestations of exocrine and endocrine pancreatic insufficiency, body mass, and QOL among patients with CP and PDAC. Materials and Methods: Prospectively collected data of 100 patients with diagnosed CP or PDAC were used for analysis. All patients underwent dual-energy X-ray absorptiometry (DXA), computed tomography (CT), and magnetic resonance imaging (MRI). The European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-C30 (EORTC QLQ-C30) was used to assess QOL. Diabetes and changes in fecal elastase-1 were also assessed. Results: There was no significant difference in skeletal muscle mass (SMM) among patients with CP and PDAC (p = 0.85). Significantly more underweight patients had low SMM (p = 0.002). Patients with CP had more pronounced pancreatic fibrosis (PF) (p < 0.001). Data showed a significant relationship between a high degree of PF and occurrence of diabetes (p = 0.006) and low fecal elastase-1 levels (p = 0.013). A statistically significant lower QOL was determined in patients with PF ≥ 50% and in the CP group. Conclusions: Sarcopenia and osteoporosis/osteopenia are highly prevalent among patients with chronic pancreatitis and pancreatic cancer, and CT- and MRI-based assessment of body composition and pancreatic fibrosis could be a potentially useful tool for routine detection of these significant metabolic changes.