Solution‐processed low‐bandgap semiconductors are crucial to next‐generation infrared (IR) detection for various applications, such as autonomous driving, virtual reality, recognitions, and quantum communications. In particular, III–V group colloidal quantum dots (CQDs) are interesting as nontoxic bandgap‐tunable materials and suitable for IR absorbers; however, the device performance is still lower than that of Pb‐based devices. Herein, a universal surface‐passivation method of InAs CQDs enabled by intermediate phase transfer (IPT), a preliminary process that exchanges native ligands with aromatic ligands on the CQD surface is presented. IPT yields highly stable CQD ink. In particular, desirable surface ligands with various reactivities can be obtained by dispersing them in green solvents. Furthermore, CQD near‐infrared (NIR) photodetectors are demonstrated using solution processes. Careful surface ligand control via IPT is revealed that enables the modulation of surface‐mediated photomultiplication, resulting in a notable gain control up to ≈10 with a fast rise/fall response time (≈12/36 ns). Considering the figure of merit (FOM), EQE versus response time (or −3 dB bandwidth), the optimal CQD photodiode yields one of the highest FOMs among all previously reported solution‐processed nontoxic semiconductors comprising organics, perovskites, and CQDs in the NIR wavelength range.