The two-photon luminescence (TPL) of small 10 nm x 40 nm colloidal gold nanorods (GNR) is investigated at the single object level, combining polarization resolved TPL and simultaneously acquired topography. A very high dependence of the TPL signal with both the nanorods longitudinal axis and the incident wavelength is observed confirming the plasmonic origin of the signal and pointing the limit of the analogy between GNRs and molecules. The spectral analysis of the TPL evidences two emission bands peaks: in the visible (in direct connection with the gold band structure), and in the infrared. Both bands are observed to vary quadradically with the incident excitation beam but exhibit different polarization properties. The maximum two-photon brightness of a single GNR is measured to be a few millions higher than the two-photon brightness of fluorescein molecules. We show that the important TPL observed in these small gold nanorods results from resonance effects both at the excitation and emission level : local field enhancement at the longitudinal surface plasmon resonances (LSPR) first results in an increase of the electronhole generation. Further relaxation of electron-hole pairs then mostly leads to the excitation of the GNR transverse plasmon mode and its subsequent radiative relaxation.