Retinoid-related orphan receptor alpha (RORα) is involved in tumor development. However, the mechanisms underlying RORα inhibiting epithelial-to-mesenchymal transition (EMT) and invasion are poorly understood in gastric cancer (GC). This study revealed that the decreased expression of RORα is associated with GC development, progression, and prognosis. RORα suppressed cell proliferation, EMT, and invasion in GC cells through inhibition of the Wnt/β-catenin pathway. RORα overexpression resulted in the decreased Wnt1 expression and the increased RORα interaction with β-catenin, which could lead to the decreased intranuclear β-catenin and p-β-catenin levels, concomitant with downregulated T-cell factor-4 (TCF-4) expression and the promoter activity of c-Myc. The inhibition of Wnt/β-catenin pathway was coupled with the reduced expression of Axin, c-Myc, and c-Jun. RORα downregulated vimentin and Snail and upregulated E-cadherin protein levels in vitro and in vivo. Inversely, knockdown of RORα attenuated its inhibitory effects on Wnt/β-catenin pathway and its downstream gene expression, facilitating cell proliferation, EMT, migration, and invasion in GC cells. Therefore, RORα could play a crucial role in repressing GC cell proliferation, EMT, and invasion via downregulating Wnt/β-catenin pathway.