A fundamental understanding of hot carrier relaxation in metal nanostructures is essential for realizing their application potential in energy conversion and photocatalysis. Despite previous investigations of the relaxation of hot carriers generated by surface plasmon resonance (SPR) excitation and interband transitions (IBTs), the hot carrier relaxation lifetimes and their associated mechanisms remain unclear. Herein, we demonstrate two distinct hot carrier relaxation channels in gold plasmonic nanostructures. The experimental observations reveal that the hot carrier relaxation is faster following SPR excitation than that from IBTs in gold nanoparticles and nanorods. The experimental results and theoretical calculations indicate that the numerous plasmon-induced hot carriers undergo surface-mediated carrier−carrier scattering in large gold nanostructures, whereas almost all IBT-induced hot carriers experience bulk carrier−carrier scattering. These findings advance our understanding of hot carrier relaxation and contribute to a clearer microscopic description of scattering channels in plasmonic nanostructures.