Photodetectors operating across the near‐ to short‐wave infrared (NIR–SWIR, λ = 0.9–1.8 µm) underpin modern science, technology, and society. Organic photodiodes (OPDs) based on bulk‐heterojunction (BHJ) active layers overcome critical manufacturing and operating drawbacks inherent to crystalline inorganic semiconductors, offering the potential for low‐cost, uncooled, mechanically compliant, and ubiquitous infrared technologies. A constraining feature of these narrow bandgap materials systems is the high noise current under an applied bias, resulting in specific detectivities (D*, the figure of merit for detector sensitivity) that are too low for practical utilization. Here, this study demonstrates that incorporating wide‐bandgap insulating polymers within the BHJ suppresses noise by diluting the transport and trapping sites as determined using capacitance‐frequency analysis. The resulting D* of NIR–SWIR OPDs operating from 600–1400 nm under an applied bias of −2 V is improved by two orders of magnitude, from 108 to 1010 Jones (cm Hz1/2 W−1), when incorporating polysulfone within the blends. This broadly applicable strategy can reduce noise in IR‐OPDs enabling their practical operation and the realization of emerging technologies.