Improved charge generation via fast and effective hole transfer in all-polymer solar cells (all-PSCs) with large highest occupied molecular orbital (HOMO) energy offset (ΔE H ) is revealed utilizing ultrafast transient absorption (TA)spectroscopy. Blending the same nonfullerene acceptor poly{[N,N′-bis(2octyldodecyl)-naphthalene-1,4,5,8-bis(dicarboximide)-2,6-diyl]-alt-5,5′-(2,2′bithiophene) (N2200) with three different donor polymers produces all-polymer blends with different ΔE H . The selective excitation of N2200 component in blends enables to uncover the hole transfer process from hole polaron-induced bleaching and absorption signals probed at different wavelength. As the ΔE H is enhanced from 0.14 to 0.37 eV, the hole transfer rate rises more than one order and the hole transfer efficiency increases from 12.9% to 86.8%, in agreement with the trend of internal quantum efficiency in the infrared region where only N2200 has absorption. Additionally, Grazing-incidence wide-angle X-ray scattering measurements indicate that face-on crystal orientation in both polymer donor and acceptor also plays an important role in facilitating the charge generation via hole transfer in all-PSCs. Hence, large ΔE H and proper crystal orientation should be considered in material design for efficient hole transfer in N2200-based heterostructures. These results can provide valuable guidance for fabrication of all-PSCs to further improve power conversion efficiency.