Interplay between strain and size quantization in a class of topological insulators based on inverted-band semiconductors

We consider surface states in semiconductors with inverted-band structures, such as $\ensuremath{\alpha}$-Sn and HgTe. The main interest is the interplay of the effect of a strain of an arbitrary sign and that of the sample finite size. We consider, in particular, a model system composed of a gapless semiconductor (e.g., HgTe or $\ensuremath{\alpha}$-Sn) of finite-width sandwiched between layers of a regular-band semiconductor (e.g., CdTe or InSb). We clarify the origin of various transitions that happen at a given strain with the change of the sample thickness, in particular the transition between the Dirac semimetal and quasi-three-dimensional (quantized) topological insulator. Our conclusion opposes those reached recently by the majority of researchers. We show that near the transition point the surface state cannot be treated as a truly topological one since the parameters of the problem are such that an appreciable overlap of the surface states' wave functions located at opposite boundaries occur. As a result, a spin-conserving, elastic impurity scattering between the states located at opposite boundaries will induce substantial backscattering and destroy the robustness of the surface state. For the k-p Kane model we derive hard-wall boundary conditions in the case when the regular-band materials form high barriers for the carriers of the inner inverted-band semiconductor (for example, the CdTe/HgTe/CdTe and $\mathrm{CdTe}/\ensuremath{\alpha}\text{\ensuremath{-}}\mathrm{Sn/CdTe}$ cases). We show that in this case the boundary conditions have universal and simple form and allow investigation of the realistic case of finite mass of the heavy-hole band, and comparison of the results obtained within the Kane and Luttinger models. In particular, a new type of surface states (wing states) developes with application of strain in the Kane model and is absent in the Luttinger model.