Context. Previous studies have discovered a population of small granules with diameters less than 800 km located in the intergranular lanes showing differing physical properties. High resolution simulations and observations of the solar granulation, in combination with automated segmentation and temporal tracking algorithms, allow us to study the evolution of the structural and physical properties of these granules and surrounding vortex motions with high temporal and spatial accuracy. Aims. We focus on the dynamics of granules, that is, the lifetime of granular cells, the fragmentation behavior, the variation of size, position, emergent intensity and vertical velocity over time and the influence of strong vortex motions. Of special interest are the dynamics of small granules compared to regular-sized granules. Methods. We developed a temporal tracking algorithm based on our previously developed segmentation algorithm for solar granulation. This was applied to radiation hydrodynamics simulations and high resolution observations of the quiet Sun by SUNRISE/IMaX. Results. The dynamics of small granules differ in regard to their diameter, intensity and depth evolution compared to the population of regular granules. The tracked granules in the simulation and observations reveal similar dynamics regarding their lifetime, evolution of size, vertical velocity and intensity. The fragmentation analysis shows that the majority of granules in the simulations do not fragment, while the opposite was found in the observations. Strong horizontal and vertical vortex motions were detected at the location of small granules. Compared to granules, regions of strong vertical vorticity show higher intensities and higher downflow velocities, and live up to several minutes. Conclusions. The analysis of granules separated according to their diameter in different groups reveals strongly differing behaviors. The largest discrepancies can be found within the groups of small, medium-sized and large granules. Therefore, these groups have to be analyzed independently. The predominant location of vortex motions on and close to small granules indicates a strong influence on the dynamics of granules.