Background: Increasing numbers of long-term gastrointestinal (GI) cancer survivors highlight the importance of understanding the factors contributing to their health-related quality of life (HRQoL). We investigated the risk factors of HRQoL, including demographics, clinical characteristics, and social and behavioral determinants of health (SBDH). Methods: Data on adult GI cancer survivors (n = 3201) from the Behavioral Risk Factors Surveillance System (BRFSS) surveys from 2014–2021 (except for 2015) were analyzed. Unadjusted/adjusted logistic regression was used. Results: The majority were women (54%) and white (78%), with a median age of 67. Survivors who were 65 years or older, diagnosed with colorectal cancer, or who had fewer comorbidities were more likely to report significantly better HRQoL. Significant social factors of poor HRQoL included unmarried, racial and ethnic minorities, poor socioeconomic status, and poor healthcare access. Significant behavioral factors of poor HRQoL were lack of physical activity, heavy alcohol consumption, and current smoking, with lack of physical activity being the most significant factor. Conclusions: The SBDH has a critical role in HRQoL. Future studies are warranted to develop a tailored survivorship intervention, such as physical rehabilitation, and to explore machine learning/artificial intelligence-based predictive models to identify cancer survivors at a high risk of developing poor HRQoL.