The luminescent ligand protected metal clusters have attracted considerable attentions while the origin of the emission still remains elusive. As recently reported in our previous work, the rod-shaped Au 25 cluster possesses a low photoluminescence quantum yield (QY=0.1%), whereas substituting silver atoms for central gold atom in the rod-shaped Au 25 cluster can drastically enhance the photoluminescence with high quantum yield (QY=40.1%). To explore the enhancement mechanism of fluorescence, femtosecond transient absorption spectroscopy is performed to determine the electronic structure and ultrafast relaxation dynamics of the highly luminescent silver-doped Ag x Au 25-x cluster by comparing the excited state dynamics of doped and un-doped Au 25 rod cluster, it is found that the excited state relaxation in Ag x Au 25-x , is proceeded with an ultrafast (~0.58 ps) internal conversion and a subsequent nuclear relaxation (~20.7 ps) followed by slow (7.4 µs) decay back to the ground state. Meanwhile, the observed nuclear relaxation is much faster in Ag x Au 25-x (~20.7 ps) compared to that in un-doped Au 25 rod (~52 ps). We conclude that it is the central Ag atom which stabilizes the charges on LUMO orbital and enhances the rigidity of Ag x Au 25-x cluster that leads to strong fluorescence.Meanwhile, coherent oscillations around ~ 0.8 THz were observed in both clusters, indicating the symmetry preservation from Au cluster to Ag alloying Au clusters. The present results provide new insights for the structure-related excited state behaviors of luminescent ligand protected Ag alloying Au clusters.