The photovoltaic characteristics of an amorphous polymer containing EDOT and fluorene units were investigated. In particular, the effects of the terminal structure, residual amount of Pd, and molecular weight were systematically investigated. Direct arylation polycondensation of EDOT followed by an established purification method readily afforded polymers with different terminal structures, Pd contents, and molecular weights. Of these factors, the terminal structure of the polymer was a crucial factor affecting the photovoltaic characteristics. For example, the polymer with a Br terminal had a PCE of 2.9% in bulk-heterojunction organic photovoltaics (BHJ OPVs) with a fullerene derivative, whereas the polymer without a Br terminal had a PCE of 4.6% in the same cell configuration. The decreased Pd residues and high molecular weights of the polymers increased the long-term stability of the devices. Moreover, BHJ OPVs containing the high-molecular-weight polymer could be fabricated with an environmentally friendly nonhalogenated solvent.