Fluorine (F) substitution on conjugated polymers in polymer solar cells (PSCs) has a diverse effect on molecular properties and device performance. We present a series of three D-A type conjugated polymers (PBT, PFBT, and PDFBT) based on dithienothiophene and benzothiadiazole units with different numbers of F atoms to explain the influence of F substitution by comparing the molecular interactions of the polymers and the recombination kinetics in PSCs. The preaggregation behavior of PFBT and PDFBT in o-DCB at the UV-vis absorption spectra proves that both polymers have strong intermolecular interactions. Besides, more closely packed structures and change into face-on orientation of fluorinated polymers are observed in polymer:PCBM blends by GIXD which is beneficial for charge transport and, ultimately, for current density in PSCs (4.3, 13.0, and 14.5 mA cm for PBT, PFBT, and PDFBT, respectively). Also, the introduction of F atoms on conjugated backbones affects the recombination kinetics by suppressing bimolecular recombination, thereby improving the fill factor (0.41, 0.68, and 0.69 for PBT, PFBT, and PDFBT, respectively). Consequently, the PCE of PSCs reached 7.3% without any additional treatment (annealing, solvent additive, etc.) in the polymer containing difluorinated BT (PDFBT) that is much higher than nonfluorinated BT (PBT ∼ 1%) and monofluorinated BT (PFBT ∼ 6%).