The electrons in the edge channels of two-dimensional topological insulators can be described as a helical Tomonaga-Luttinger liquid. They couple to nuclear spins embedded in the host materials through the hyperfine interaction, and are therefore subject to elastic spin-flip backscattering on the nuclear spins. We investigate the nuclear-spin-induced edge resistance due to such backscattering by performing a renormalization-group analysis. Remarkably, the effect of this backscattering mechanism is stronger in a helical edge than in nonhelical channels, which are believed to be present in the trivial regime of InAs/GaSb quantum wells. In a system with sufficiently long edges, the disordered nuclear spins lead to an edge resistance which grows exponentially upon lowering the temperature. On the other hand, electrons from the edge states mediate an anisotropic Ruderman-Kittel-Kasuya-Yosida nuclear spin-spin interaction, which induces a spiral nuclear spin order below the transition temperature. We discuss the features of the spiral order, as well as its experimental signatures. In the ordered phase, we identify two backscattering mechanisms, due to charge impurities and magnons. The backscattering on charge impurities is allowed by the internally generated magnetic field, and leads to an Anderson-type localization of the edge states. The magnon-mediated backscattering results in a power-law resistance, which is suppressed at zero temperature. Overall, we find that in a sufficiently long edge the nuclear spins, whether ordered or not, suppress the edge conductance to zero as the temperature approaches zero.