Since the discovery of electron-wave duality, electron scattering instrumentation has developed into a powerful array of techniques for revealing the atomic structure of matter. Beyond detecting local lattice variations in equilibrium structures with the highest possible spatial resolution, recent research efforts have been directed towards the long sought-after dream of visualizing the dynamic evolution of matter in real-time. The atomic behavior at ultrafast timescales carries critical information on phase transition and chemical reaction dynamics, the coupling of electronic and nuclear degrees of freedom in materials and molecules, the correlation between structure, function and previously hidden metastable or nonequilibrium states of matter. Ultrafast electron pulses play an essential role in this scientific endeavor, and their generation has been facilitated by rapid technical advances in both ultrafast laser and particle accelerator technologies. This review presents a summary of the remarkable developments in this field over the last few decades. The physics and technology of ultrafast electron beams is presented with an emphasis on the figures of merit most relevant for ultrafast electron diffraction (UED) experiments. We discuss recent developments in the generation, manipulation and characterization of ultrashort electron beams aimed at improving the combined spatio-temporal resolution of these measurements. The fundamentals of electron scattering from atomic matter and the theoretical frameworks for retrieving dynamic structural information from solid-state and gas-phase samples is described. Essential experimental techniques and several landmark works that have applied these approaches are also highlighted to demonstrate the widening applicability of these methods. Ultrafast electron probes with ever improving capabilities, combined with other complementary photonbased or spectroscopic approaches, hold tremendous potential for revolutionizing our ability to observe and understand energy and matter at atomic scales.