The separation of Ge and Si by an electrically isolating dielectric layer is essential to yield high efficiency for optical telecommunication applications and electronic applications such as Ge MOSFETs. Ge epitaxial lateral overgrowth (ELOG) is a promising approach to achieve Ge on Si separated by a thin dielectric layer. However, a general understanding of the anisotropic dynamics of ELOG Ge on Si is limited, which prevents its wide adoption. In this paper, we report how the orientation and width of the dielectric layer controls the ELOG. A competitive ELOG from perpendicular directions on a dielectric strip leads to a rapid growth along the long axis of the dielectric layer, or a mixed coalescence from perpendicular directions yielding various Ge confined configurations at the Ge/dielectric-layer interface. Especially, an angle of 7.5° between dielectric-layer and Si [110] axis shows the most pronounced unidirectional ELOG. ELOG disappears as the width of the dielectric mask exceeds 5.0 μm. The results reported here provide a general framework for ELOG of semiconductor materials.