Currently, the performance improvement for inverted perovskite solar cells (PVSCs) is mainly limited by the high open circuit voltage (VOC) loss caused by detrimental non‐radiative recombination (NRR) processes. Herein, we report a simple and efficient way to simultaneously reduce the NRR processes inside perovskites and at the interface by rationally designing a new pyridine‐based polymer hole‐transporting material (HTM), PPY2, which exhibits suitable energy levels with perovskites, high hole mobility, effective passivation of the uncoordinated Pb2+ and iodide defects, as well as the capability of promoting the formation of high‐quality polycrystalline perovskite films. In absence of any dopants, the inverted PVSCs using PPY2 as the HTM deliver an encouraging PCE up to 22.41 % with a small VOC loss (0.40 V), among the best device performances for inverted PVSCs reported so far. Furthermore, PPY2‐based unencapsulated devices show an excellent long‐term photostability, and over 97 % of its initial PCE can be maintained after one sun constant illumination for 500 h.