Connecting the collective behavior of disordered systems with local structure on the particle scale is an important challenge, for example, in granular and glassy systems. Compounding complexity, in many scientific and industrial applications, particles are polydisperse, aspherical, or even of varying shape. Here, we investigate a generalization of the classical kissing problem in order to understand the local building blocks of packings of aspherical grains. We numerically determine the densest local structures of uniaxial ellipsoids by minimizing the Set Voronoi cell volume around a given particle. Depending on the particle aspect ratio, different local structures are observed and classified by symmetry and Voronoi coordination number. In extended disordered packings of frictionless particles, knowledge of the densest structures allows us to rescale the Voronoi volume distributions onto the single-parameter family of k-Gamma distributions. Moreover, we find that approximate icosahedral clusters are found in random packings, while the optimal local structures for more aspherical particles are not formed.