A comprehensive morphological stability analysis of a nanoscale circular island during heteroepitaxial growth is presented based on continuum elasticity theory. The interplay between kinetic and thermodynamic mechanisms is revealed by including strain-related kinetic processes. In the kinetic regime, the Burton-Cabrera-Frank model is adopted to describe the growth front of the island. Together with kinetic boundary conditions, various kinetic processes including deposition flow, adatom diffusion, attachment-detachment kinetics, and the Ehrlich-Schwoebel barrier can be taken into account at the same time. In the thermodynamic regime, line tension, surface energy, and elastic energy are considered. As the strain relief in the early stages of heteroepitaxy is more complicated than commonly suggested by simple consideration of lattice mismatch, we also investigate the effects of external applied strain and elastic response due to perturbations on the island shape evolution. The analytical expressions for elastic fields induced by mismatch strain, external applied strain, and relaxation strain are presented. A systematic approach is developed to solve the system via a perturbation analysis which yields the conditions of film morphological instabilities. Consistent with previous experimental and theoretical work, parametric studies show the kinetic evolution of elastic relaxation, island morphology, and film composition under various conditions. Our present work offers an effective theoretical approach to get a comprehensive understanding of the interplay between different growth mechanisms and how to tailor the growth mode by controlling the nature of the crucial factors. V C 2015 AIP Publishing LLC.