Despite mounting evidence that materials imperfections are a major obstacle to practical applications of superconducting qubits, connections between microscopic material properties and qubit coherence are poorly understood. Here, we perform measurements of transmon qubit relaxation times T1 in parallel with spectroscopy and microscopy of the thin polycrystalline niobium films used in qubit fabrication. By comparing results for films deposited using three techniques, we reveal correlations between T1 and grain size, enhanced oxygen diffusion along grain boundaries, and the concentration of suboxides near the surface. Physical mechanisms connect these microscopic properties to residual surface resistance and T1 through losses arising from the grain boundaries and from defects in the suboxides. Further, experiments show that the residual resistance ratio can be used as a figure of merit for qubit lifetime. This comprehensive approach to understanding qubit decoherence charts a pathway for materials-driven improvements of superconducting qubit performance.