The permanent strengths of interstitial-free (IF) steels with different grain sizes and dislocation densities processed by severe plastic deformation (SPD) and subsequent annealing are systematically investigated. Permanent strength, which is athermal and time-independent, corresponds to the fundamental capability to bear stresses caused by external forces. Sufficiently long-time (24 h) stress relaxation tests were carried out and experimental stress-relaxation time relationships were extrapolated to estimate the permanent strength that remained after an infinite time passed. The flow stresses observed in standard uniaxial tension tests increased with repeated SPD processes, and the fraction of permanent strength to the observed flow stress was mostly above 65%. The permanent strength also increased with repetition of SPD processes, and subsequent low-temperature annealing further augmented the permanent strength. During SPD processes, the dislocation-related strengthening was dominant, while the grain-size-related strengthening was minor, i.e., the Hall-Petch relation does not hold. On the other hand, after low-temperature annealing, the grain-sizerelated strengthening became dominant, quickly replacing the dislocation-related strengthening. In a coarse-grain region, the grain-size-related strength was consistent with the classical Hall-Petch relation. It was confirmed that the original Hall-Petch relation holds only in the coarse-grain region and it indicates "softening with grain coarsening due to annealing", not "strengthening by grain refinement due to SPD".