We present a method to microscopically derive a small-size k·p Hamiltonian in a Hilbert space spanned by physically chosen ab initio spinor wave functions. Without imposing any complementary symmetry constraints, our formalism equally treats three-and two-dimensional systems and simultaneously yields the Hamiltonian parameters and the true Z2 topological invariant. We consider bulk crystals and thin films of Bi2Se3, Bi2Te3, and Sb2Te3. It turns out that the effective continuous k·p models with open boundary conditions often incorrectly predict the topological character of thin films.PACS numbers: 71.18.+y, 71.70.Ej, Electronic structure of topological insulators (TIs) has been in focus of theoretical research regarding linear response, transport properties, Hall conductance, and motion of Dirac fermions in external fields [1,2]. These problems call for a physically justified model Hamiltonian of small dimension. As in semiconductors, it is thought sufficient that the model accurately reproduces the TI band structure near the inverted band gap [3]. The desired Hamiltonian is derived either from the theory of invariants [4] or within the k·p perturbation theory using the symmetry properties of the basis states [5].In Ref.[3], along with the pioneering prediction of the topological nature of Bi 2 Se 3 , Bi 2 Te 3 , and Sb 2 Te 3 , a 4-band Hamiltonian was first constructed from the theory of invariants, which is presently widely used to analyze the properties of bulk TIs as well as their surfaces and thin films [6][7][8][9][10][11][12][13][14]. The Hamiltonian parameters in Ref. [3] were obtained by fitting ab initio band dispersion curves. Later, an attempt was made [15] to recover the Hamiltonian of Ref.[3] by a k·p perturbation theory with symmetry arguments and to derive its parameters from the ab initio wave functions of the bulk crystals. Furthermore, in Ref. To analyze how the properties of thin films are inherited from the bulk TI features, effective continuous models have been developed: they are based on the substitution k z → −i∂ z (originally introduced for slowly varying perturbations [16]) in the Hamiltonian of Ref.[3] and on the imposition of the open boundary conditions [15,[17][18][19]. These models predict a variety of intriguing phenomena at surfaces, interfaces, and thin films of TIs [20][21][22][23]. A fundamental issue here is the topological phase transition between an ordinary 2D insulator and a quantum spin Hall insulator (QSHI). Apart from the theoretical prediction, the model parameters are fitted to the measured band dispersion to deduce the topological phase from the experiment [24,25]. By analyzing the signs and relative values of the parameters of the empirically obtained effective model a judgement is made on whether the edge states would exist in a given TI film, the logic being similar to that of Ref. [26]: The valence band should have a positive and conduction band a negative effective mass.In order to avoid any ambiguity in deriving the model Hamiltonian and to treat 3D and 2D s...