By integrating full‐duplex (FD) and nonorthogonal multiple‐access (NOMA) techniques for wireless self‐backhaul at small‐cell base stations (SBSs), a novel multitier heterogeneous network (HetNet) is proposed. In the HetNet, NOMA is used at SBSs for combing backhaul and downlink (DL) signals. As a result, at each SBS, the backhaul, uplink transmission, and DL transmission are executed simultaneously, whereas the FD macro base stations (MBSs) communicate with mobile users (MUs) and transceive backhaul signals from associated SBSs by using massive‐MIMO antennas. The simple time‐division duplex is performed for the communications between MBSs and MUs so that a round of communications consists of MBS‐MU‐DL and MBS‐MU‐UL phases. This work particularly focuses on the MBS‐MU‐DL phase. By assuming the HetNet to be fully loaded, this paper first models the distribution of active SBSs and those of the interference received at the different types of receivers. Then, by formulating the received signal‐to‐interference ratios, this paper obtains the coverage probabilities of the MU‐SBS‐MBS uplink backhaul and MU DLs as well as the spectrum and energy efficiencies. The presented simulations and numerical results show that the integration of FD and NOMA at SBSs is an effective solution for creating the backhaul between FD‐SBS and FD‐MBS. It is obtained that when the power‐sharing coefficient allocated for a small‐cell DL is small, the coverage probability of SBS DL transmission increases; otherwise, it decreases. However, the coverage probabilities of the MBS DL and MU‐SBS‐MBS backhaul link monotonically vary with the power‐sharing coefficient. It is also achieved that the power‐sharing coefficient imposes the effect on the energy efficiencies of the MU‐SBS‐MBS backhaul link and MU DL in inverse directions. At the same time, it is found that the achieved energy efficiency gain depends greatly on the resolution of massive‐MIMO antennas.