To investigate the effect of symmetry of copolymerized unit on the properties of naphthalene diimide (NDI)‐based photovoltaic polymers, three n‐type semiconductor polymers, PNDI‐BDT, PNDI‐T‐BTh, and PNDI‐T‐P‐T are synthesized by copolymerizing NDI with benzodithiophene (BDT), thienylene‐benzothiophene (T‐BTh), and thienylene‐phenylene‐thienylene (T‐P‐T) units, respectively. The latter two building blocks could be regarded as the derivatives from BDT by molecular tailoring and only T‐BTh is an asymmetric segment. The light absorption, carrier mobility, film morphology, crystallinity, and photovoltaic properties of three PNDIs are systematically characterized and compared. Although PNDI‐T‐BTh adopts a face‐on molecular orientation, it shows a high field‐effect electron mobility (μe) of 1.06 × 10−2 cm2 V−1 s−1, which is comparable to the μe of PNDI‐BDT (1.37 × 10−2 cm2 V−1 s−1) with edge‐on molecular orientation. The photovoltaic devices with PNDI‐T‐BTh as the acceptor show the highest power conversion efficiency (PCE) of 5.99% with VOC of 0.87 V, JSC of 10.28 mA cm−2, and FF of 0.67, while PNDI‐BDT and PNDI‐T‐P‐T containing the symmetric copolymerized units give lower PCE of 2.06 and 4.12%, respectively. The higher PCE with PNDI‐T‐BTh could be attributed to the favorable face‐on molecular orientation, fiber‐like blend film morphology and relatively balanced hole and electron mobilities. Our results indicate that the introduction of the asymmetric unit could efficiently improve the photovoltaic performance, which provides a feasible and effective strategy to design novel NDI‐based and even other type rylene diimide‐based photovoltaic polymers.