Optical properties of multi-stacked InGaAs/GaNAs quantum dot solar cell fabricated on GaAs (311)B substrate J. Appl. Phys. 112, 064314 (2012) Optical polarization properties of a nanowire quantum dot probed along perpendicular orientations Appl. Phys. Lett. 101, 111112 (2012) Electroluminescence from quantum dots fabricated with nanosphere lithography Appl. Phys. Lett. 101, 103105 (2012) Nucleation features and energy levels of type-II InAsSbP quantum dots grown on InAs(100) substrate Appl. Phys. Lett. 101, 093103 (2012) Highly luminescing multi-shell semiconductor nanocrystals InP/ZnSe/ZnS Appl. Phys. Lett. 101, 073107 (2012) Additional information on J. Appl. Phys. This article presents the results of three-dimensional modeling of heteroepitaxial thin film growth with the objective of understanding recent experiments on the early stages of quantum dot formation in SiGe/Si systems. We use a continuum model, based on the underlying physics of crystallographic surface steps, to study the growth of quantum dots, their spatial ordering and coarsening behavior. Using appropriate parameters, obtained from atomistic calculations, the ͑100͒ orientation is found to be unstable under compressive strains. The surface energy now develops a minimum at an orientation that may be interpreted as the ͑105͒ facet observed in SiGe/Si systems. This form of the surface energy allows for the growth of quantum dots without any barrier to nucleation-dots are seen to start off via a surface instability as shallow stepped mounds, which steepen continuously to reach their low energy orientations. During the very initial stages of growth, mounds are seen to grow in a dense array with several of them impinging on each other and subsequently coalescing to form larger mounds. This behavior occurs due to the competition between surface energy which seeks to minimize the free-energy by the formation of islands with side-walls at the strain stabilized orientations and repulsive elastic interactions between such closely spaced islands. Using simple analytical calculations, we show the existence of a critical island size for this coalescence behavior. A key result of our analysis is the inverse scaling of this critical size with the misfit strain in the film. While energetic analyses may be used to obtain useful insights, the growth of quantum dots is essentially a nonequilibrium process and requires a fundamental understanding of the kinetics. Numerical studies show that the growth kinetics has a profound effect on surface morphology: arrays of well-separated islands or, alternatively, intersecting ridges are obtained in different kinetic regimes. We also study an alternative model of a stable but nonfacet ͑100͒ orientation and point out the inconsistencies of this assumption.