The aggregation behavior of polymers can ultimately have a significant impact on the performance of polymer photodetectors depends mainly on the main solvent, solvent additive, thermal annealing treatment, and so on. In this work, a phase control behavior of lateral polymer photodetectors (L‐PPDs) is realized by changing the acceptor concentration or introducing 1,8‐diiodooctane (DIO) in strong aggregation of bulk heterojunction (BHJ) films, which can effectively regulate the minority carriers trapping under high PC61BM ratio. Additionally the responsivity (R), gain (G) and specific detectivity (D*) value of the L‐PPDs can reach 94.4 A W−1, 289.7, and 1.33 × 1014 Jones under 0.0031 mW cm−2 @405 nm, respectively. The structure of the L‐PPDs is optimized as quartz/ PMMA (30 nm)/ PffBT4T‐2OD:PC61BM (D/A ratio = 1:1.2)/ Ag–Ag electrodes. Furthermore, the phase separation of the thin films is promoted by introducing 3% DIO, resulting in a clear increase of response speed (fall time is about tenfold faster than that of the control device), and an improved switching performance about 60‐fold higher under strong light (10.42 mW cm−2). This research represents an important step forward in the fabrication of high‐performance polymer photodetectors, which show promising application potential.