Low temperature epitaxial breakdown of inhomogeneously strained Si capping layers is investigated. By growing Si films on coherently strained GeSi quantum dot surfaces, we differentiate effects of surface roughness, strain, and growth orientation on the mechanism of epitaxial breakdown. Using atomic force microscopy and high resolution cross-sectional transmission electron microscopy we find that while local lattice strain up to 2% has a negligible effect, growth on higher-index facets such as {113} significantly reduces the local breakdown thickness. Nanoscale growth mound formation is observed above all facet orientations. Since diffusion lengths depend directly on the surface orientation, we relate the variation in epitaxial thickness to low temperature stability of specific growth facets and on the average size of kinetically limited growth mounds.