Epitaxial growth is a fundamental step required to create devices for the semiconductor industry, enabling different materials to be combined in layers with precise control of strain and defect structure. Patterning the growth substrate with a mask before performing epitaxial growth offers additional degrees of freedom to engineer the structure and hence function of the semiconductor device. Here, we demonstrate that conditions exist where such epitaxial lateral overgrowth can produce complex, three-dimensional structures that incorporate cavities of deterministic size. We grow germanium on silicon substrates patterned with a dielectric mask and show that fully-enclosed cavities can be created through an unexpected self-assembly process that is controlled by surface diffusion and surface energy minimization. The result is confined cavities enclosed by single crystalline Ge, with size and position tunable through the initial mask pattern. We present a model to account for the observed cavity symmetry, pinch-off and subsequent evolution, reflecting the dominant role of surface energy. Since dielectric mask patterning and epitaxial growth are compatible with conventional device processing steps, we suggest that this mechanism provides a strategy for developing electronic and photonic functionalities.