Esophageal, esophago-gastric, and gastric cancers are major causes of cancer morbidity and cancer death. For patients with potentially resectable disease, multimodality treatment is recommended as it provides the best chance of survival. However, quality of life may be adversely affected by therapy, and with a wide variation in outcome despite multi-modality therapy, there is a clear need to improve patient stratification. Radiomic approaches provide an opportunity to improve tumor phenotyping. In this review we assess the evidence to date and discuss how these approaches could improve outcome in esophageal, esophago-gastric, and gastric cancer.