To establish molecular design rules for the donor polymers matching well with both fullerene and nonfullerene acceptors in polymer solar cells (PSCs), we investigated the effect of polymer aggregation on the donor/acceptor blend morphology and consequent device performances by comparing the femtosecond transient absorption behaviors of two different donor polymers with a virtually identical backbone structure. The donor polymers, PBDCS and PBDS, have almost the same polymer backbone structure of which aggregation properties however differ significantly depending on the presence of an aggregation promotor (β-cyano groups in PBDCS). Furthermore, a fullerene acceptor, PC 71 BM, and a nonfullerene acceptor, ITIC, also with different aggregation and percolation properties were selected. Therefore, we could make a two donor−two acceptor pairing system to explore the difference in blend morphology with their different aggregation properties and miscibility between donor and acceptor molecules. Due to its strong aggregation properties and limited miscibility with acceptors, PBDCS-based PSC devices exhibited delayed charge generation in the pure domain system, suppressed charge recombination, and longer-lived polarons leading to the efficient charge extraction and high photovoltaic performances. In contrast, PBDS exhibited largely different photovoltaic performances in the PSCs using PC 71 BM and ITIC because of the nonaggregating property of PBDS and different blend morphologies. In the excessively intermixed system, charge generation is ultrafast and very efficient, but increased charge recombination and shorter-lived polarons reduce charge transport capability, resulting in poor photovoltaic performances. This work reports a comprehensive guideline how to predict and obtain high-efficiency photovoltaic characteristics by rationalizing the charge generation and recombination processes through combined spectroscopic and photovoltaic studies.