Recent discoveries have spurred the theoretical prediction and experimental realization of novel materials that have topological properties arising from band inversion. Such topological insulators are insulating in the bulk but have conductive surface or edge states. Topological materials show various unusual physical properties and are surmised to enable the creation of exotic Majorana-fermion quasiparticles. How the signatures of topological behavior evolve when the system size is reduced is interesting from both a fundamental and an application-oriented point of view, as such understanding may form the basis for tailoring systems to be in specific topological phases. This work considers the specific case of quantum-well confinement defining two-dimensional layers. Based on the effective-Hamiltonian description of bulk topological insulators, and using a harmonic-oscillator potential as an example for a softer-than-hard-wall confinement, we have studied the interplay of band inversion and size quantization. Our model system provides a useful platform for systematic study of the transition between the normal and topological phases, including the development of band inversion and the formation of massless-Dirac-fermion surface states. The effects of bare size quantization, twodimensional-subband mixing, and electron-hole asymmetry are disentangled and their respective physical consequences elucidated.