The ability to predict times of greater galactic cosmic ray (GCR) fluxes is important for reducing the hazards caused by these particles to satellite communications, aviation, or astronauts. The 11-year solar cycle variation in cosmic rays is highly correlated with the strength of the heliospheric magnetic field. Differences in GCR flux during alternate solar cycles yield a 22-year cycle, known as the Hale Cycle, which is thought to be due to different particle drift patterns when the northern solar pole has predominantly positive (denoted a qA>0 cycle) or negative (qA<0) polarities. This results in the onset of the peak cosmic ray flux at Earth occurring earlier during qA>0 cycles than for qA<0 cycles and hence the peak being more domed for qA>0 and more sharply peaked for qA<0. In this study, we demonstrate that properties of the large-scale heliospheric magnetic field are different during the declining phase of the qA<0 and qA>0 solar cycles, when the difference in GCR flux is most apparent. This suggests that particle drifts may not be the sole mechanism responsible for the Hale Cycle in GCR flux at Earth. However, it is also demonstrated that these polarity-dependent heliospheric differences are evident during the space-age but much less clear in earlier data: using geomagnetic reconstructions, it is shown that for the period of 1905 -1965, alternate polarities do not give as significant a difference during the declining phase of the solar cycle. Thus we suggest that the 22-year cycle in cosmic ray flux is at least partly the result of direct modulation by the heliospheric magnetic field and that this effect may be primarily limited to the grand solar maximum of the space-age.