Using scanning tunneling microscopy supplemented with first-principles calculations, we examine all the thermally activated atomistic processes of Ag n (n = 1-26) constructed atom-by-atom on a Si(111)-(7 × 7) substrate, and we exploit such cluster dynamical information to further determine the energetic stability (or magicity) of the clusters. By generalizing the traditional concept of cluster magicity solely based on cluster association/dissociation to also include various complex collective cluster motions, we identify the existence of two classes of magic clusters. The most stable class, Ag 10 and Ag 25 , is defined by geometrical shell closure; the less stable class of Ag n (n = 3, 5, 13, 16, 19) is associated with lower kinetic barriers against internal restructuring of, or atom detachment from, their respective clusters of neighboring sizes. Our detailed analysis also reveals that the substrate effect, rather than the number of bonds within the clusters, dominates the cluster stabilities. The conceptual advances gained in the present study are broadly applicable to many related cluster systems in contact with external media, and they are expected to be instrumental in tuning the dynamical behaviors of clusters in surface catalysis, nanoplasmonics, and other technological areas.