High‐performance IR organic photodetectors (OPDs) are of great significance for wireless optical communication, light detection and ranging (LiDAR) technology, and wearable electronics. However, high dark current and low responsivity (R) hinder their future commercial application. Herein, fullerene and non‐fullerene acceptors‐based OPDs are fabricated to understand the relationship between the trap density and photo‐responsivity. Impressively, the non‐fullerene system (Poly([2,6′‐4,8‐di(5‐ethylhexylthienyl)benzo[1,2‐b;3,3‐b]dithiophene]{3‐fluoro‐2[(2‐ethylhexyl)carbonyl]thieno[3,4‐b]thiophenediyl}) (PCE10):BTPV‐4F‐eC9) based OPDs exhibits a record R‐value of 0.56 A W−1 at 900 nm in no gain photodiode‐type OPDs, which results in a high detectivity over 1013 Jones in 400–1030 nm at room temperature. Mechanistic studies show that the low trap density plays critical role in reducing the trap‐assisted recombination and density of thermally generated carriers, thus improving the responsivity and reducing the dark current of the device. These findings provide new insights into the mechanism of high‐performance self‐powered near‐infrared OPDs.