The main drivers of COVID-19 disease severity and the impact of COVID-19 on long-term health after recovery are yet to be fully understood. Medical imaging studies investigating COVID-19 to date have mostly been limited to small datasets and post-hoc analyses of severe cases. The UK Biobank recruited recovered SARS-CoV-2 positive individuals (n=967) and matched controls (n=913) who were extensively imaged prior to the pandemic and underwent follow-up scanning. In this study, we investigated longitudinal changes in body composition, as well as the associations of pre-pandemic image-derived phenotypes with COVID-19 severity. Our longitudinal analysis, in a population of mostly mild cases, associated a decrease in lung volume with SARS-CoV-2 positivity. We also observed that increased visceral adipose tissue and liver fat, and reduced muscle volume, prior to COVID-19, were associated with COVID-19 disease severity. Finally, we trained a machine classifier with demographic, anthropometric and imaging traits, and showed that visceral fat, liver fat and muscle volume have prognostic value for COVID-19 disease severity beyond the standard demographic and anthropometric measurements. This combination of image-derived phenotypes from abdominal MRI scans and ensemble learning to predict risk may have future clinical utility in identifying populations at-risk for a severe COVID-19 outcome.