The influence of passivating ligand on electron-phonon relaxation dynamics of the smallest sized gold clusters was studied using ultrafast transient absorption spectroscopy and theoretical modeling. The electron dynamics in Au279, Au329, and Au329 passivated with 4-tert-butylbenzene thiol (TBBT), phenylethane thiol (SC2Ph) and hexane thiol (SC6), respectively, were investigated. These clusters were chosen as they are the smallest gold clusters reported till-date to show plasmonic behavior. Ultrafast transient absorption measurements were also carried out on Au~1400 (SC6) and Au~2000 (SC6) to understand the influence of the size on electron-phonon relaxation with the same passivating ligand. The study has revealed interesting aspects on the role of ligand on electron-phonon relaxation dynamics