Unlike in bulk materials, energy transport in low-dimensional and nanoscale systems may be governed by a coherent "ballistic" behavior of lattice vibrations, the phonons. If dominant, such behavior would determine the mechanism for transport and relaxation in various energy-conversion applications. In order to study this coherent limit, both the spatial and temporal resolutions must be sufficient for the length-time scales involved. Here, we report observation of the lattice dynamics in nanoscale quantum dots of gallium arsenide using ultrafast electron diffraction. By varying the dot size from h = 11 to 46 nm, the length scale effect was examined, together with the temporal change. When the dot size is smaller than the inelastic phonon mean-free path, the energy remains localized in high-energy acoustic modes that travel coherently within the dot. As the dot size increases, an energy dissipation toward lowenergy phonons takes place, and the transport becomes diffusive. Because ultrafast diffraction provides the atomic-scale resolution and a sufficiently high time resolution, other nanostructured materials can be studied similarly to elucidate the nature of dynamical energy localization.