While the metastable β (A15) phase of tungsten has one of the largest spin Hall angles measured, the origin of this high spin Hall conductivity is still unclear. Since large concentrations of oxygen and nitrogen are often used to stabilize β tungsten, it is not obvious whether the high spin Hall conductivity is due to an intrinsic or extrinsic effect. In this work, we have examined the influence of O and N dopants on the spin Hall conductivity and spin Hall angle of β-W. Using multiple first principles approaches, we examine both the intrinsic and extrinsic (skew scattering) contributions to spin Hall conductivity. We find that intrinsic spin Hall conductivity calculations for pristine β-W are in excellent agreement with experiment. However, when the effect of high concentrations (11 at.%) of O or N interstitials on the electronic structures is taken into account, the predicted intrinsic spin Hall conductivity is significantly reduced. Skew scattering calculations for O and N interstitials in β-W indicate that extrinsic contributions have a limited impact on the total spin Hall conductivity. However, we find that the spin-flip scattering at O and N impurities can well explain the experimentally found spin-diffusion length within the range of 1-5 nm. To explain these findings, we propose that dopants (O and N) help to stabilize β-W grains during film deposition and afterwards segregate to the grain boundaries. This process leads to films of relatively pristine small β-W grains and grain boundaries with high concentrations of O or N scattering sites. This combination provides high spin Hall conductivity and large electrical resistance, leading to high spin Hall angles. This work shows that engineering grain boundary properties in other high spin Hall conductivity materials could provide an effective way to boost the spin Hall angle.