We study the role of the drift effect in the temporal changes of the anisotropy of galactic cosmic rays (GCRs) and the influence of the sector structure of the heliospheric magnetic field on it. We analyze the GCR anisotropy in Solar Cycle 24 and solar minimum 23/24 with negative polarity (qA < 0) for the period of 2007-2009 and near minimum 24/25 with positive polarity (qA > 0) in 2017-2018 using data of the global network of Neutron Monitors. We use the harmonic analysis method to calculate the radial and tangential components of the anisotropy of GCRs for different sectors ('+' corresponds to the positive and '−' to the negative directions) of the heliospheric magnetic field. We compare the analysis of GCR anisotropy using different evaluations of the mean GCRs rigidity related to Neutron Monitor observations. Then the radial and tangential components are used for characterizing the GCR modulation in the heliosphere. We show that in the solar minimum 23/24 in 2007-2009 when qA < 0, the drift effect is not visibly evident in the changes of the radial component, i.e. the drift effect is found to produce ≈ 4% change in the radial component of the GCR anisotropy for 2007-2009. Hence the diffusion dominated model of GCR transport is more acceptable in 2007-2009. In turn, near the solar minimum 24/25 in 2017-2018 when qA > 0, the drift effect is evidently visible and produces ≈40% change in the radial component of the GCR anisotropy for 2017-2018. So in the period of 2017-2018 a diffusion model with noticeably manifested drift is acceptable. The results of this work are in good agreement with the drift theory of GCR modulation, according to which, during negative (positive) polarity cycles, a drift stream of GCRs is directed toward (away from) the Sun, thus giving rise to a 22-year cycle variation of the radial GCR anisotropy.