A series of high-performance organic semiconductors, which are modulated by introducing heteroatoms to rationally control molecular packing and charge carrier transport, have been successfully reported. However, a fundamental physical understanding of the impact of chalcogen atoms on intermolecular interactions between donors and acceptors as well as photophysical process in photovoltaic cells is still lagging. Herein, a detailed investigation on rylene imide-based solar cells is carried out to reveal the role of chalcogen atoms in controlling intermolecular interactions, molecular orientation in bulk and at the donor− acceptor interface, and polaron-pair dissociation. Compared to their Se-atom-free assisted counterparts, poly{ [4,8-bis[5-(2-ethylhexyl)-4-fluoro-2-thienyl]benzo[1,2-b:4,5-b′]dithiophene2,6diyl]-alt-[2,5-thiophenediyl[5,7-bis(2-ethylhexyl)-4,8-dioxo-4H,8H-benzo[1,2-c:4,5-c′]dithiophene-1,3-diyl]]} (PBDB-TF): selenium-annulated triperylene hexaimide (TPH-Se) bulk heterojunctions preserve face-on orientation and possess smaller domain size, which are partially attributed to the Se•••O van der Waals contacts between the acceptor and polymer chain. This feature enables PBDB-TF:TPH-Se interfaces with enhanced π-orbital overlap, improved charge transfer, a narrowed chargetransfer band, and suppressed polaron-pair binding energy. Consequently, all of the Se-containing solar cells investigated in this manuscript exhibit higher short-circuit current densities and conversion efficiencies than those in Se-atom-free devices. Our results reveal an important molecular design strategy for high-performance rylene imide-based acceptors: efficiently improving the electronic interactions at the D−A interface to increase polaron-pair dissociation and suppress geminate recombination.