Generation of hot carriers from nonradiative surface-plasmon decay and subsequent injection into the semiconductor is the start of a new paradigm in the field of optoelectronics and photocatalysis. However, the role of phonons in the plasmon-induced charge-transfer process is mainly unexplored. We utilized a technique called photoassisted scanning tunneling spectroscopy (PATS) to probe it by measuring the photoconduction of CdSe QDs deposited on gold and conducting n-type silicon. The ratio of currents in the light and dark shows much higher for CdSe QDs thin film on the gold substrate than the n-type silicon. Here, phonons impersonate an essential role in a higher generation rate and ultrafast injection of hot carriers into CdSe QDs interfaced with gold substrate. Scanning tunneling spectroscopy results recorded under the light with band-edge-like peaks located inside the band gap of CdSe QDs on gold indicate the quantum tunneling of plasmonic hot carriers to the defect levels directly, which would contribute significantly to enhance the photoresponse property of CdSe QDs.