The trapped residual magnetic flux during the cool-down due to the incomplete Meissner state is a significant source of radio frequency losses in superconducting radio frequency cavities. Here, we clearly correlate the niobium microstructure in elliptical cavity geometry and flux expulsion behavior. In particular, a traditionally fabricated Nb cavity half-cell from an annealed poly-crystalline Nb sheet after an 800 ∘C heat treatment leads to a bi-modal microstructure that ties in with flux trapping and inefficient flux expulsion. This non-uniform microstructure is related to varying strain profiles along the cavity shape. A novel approach to prevent this non-uniform microstructure is presented by fabricating a 1.3 GHz single cell Nb cavity with a cold-worked sheet and subsequent heat treatment leading to better flux expulsion after 800 ∘C/3 h. Microstructural evolution by electron backscattered diffraction-orientation imaging microscopy on cavity cutouts, and flux pinning behavior by dc-magnetization on coupon samples confirms a reduction in flux pinning centers with increased heat treatment temperature. The heat treatment temperature-dependent mechanical properties and thermal conductivity are reported. The significant impact of cold work in this study demonstrates clear evidence for the importance of the microstructure required for high-performance superconducting cavities with reduced losses caused by magnetic flux trapping.