Ramie cell walls play an important role in cadmium (Cd) detoxification. However, the Cd binding capacity of the cell wall components and the cell wall compositions among ramie species remains unclear. Therefore, this study compared two ramie populations (‘Dazhuhuangbaima’ (low-Cd-accumulating population) and ‘Zhongzhu 1’ (high-Cd-accumulating population)) with different Cd enrichment characteristics. The two ramie populations were treated with 0, 25, and 75 mg kg−1 Cd for 30 days; then, their root length, plant height, biomass, Cd enrichment in the organs, subcellular Cd distribution, Cd content in the cell wall polysaccharides, and hemicellulose content were determined. The root length, plant height, biomass, and Cd enrichment in all organs were significantly higher (p ≤ 0.05) in ‘Zhongzhu 1’ than in ‘Dazhuhuangbaima’ under Cd stress. In addition, the subcellular Cd distribution analysis revealed that Cd was mainly found in the cell wall in both ramie populations. Among the cell wall fractions, Cd was mainly bound to the hemicelluloses, with 60.38–73.10% and 50.05–64.45% Cd accumulating in the ‘Zhongzhu 1’ and ‘Dazhuhuangbaima’ cell wall hemicelluloses, respectively. However, the Cd concentration in the ‘Zhongzhu 1’ hemicellulose was significantly higher (p ≤ 0.05) than that in the ‘Dazhuhuangbaima’ hemicellulose. Hemicellulose content analysis further revealed that the hemicellulose concentration increased with the Cd concentration in both populations, but it was significantly higher (p ≤ 0.05) in ‘Zhongzhu 1’ than in ‘Dazhuhuangbaima’ across all Cd treatments. Thus, ramie copes under Cd stress by increasing the hemicellulose content in the cell wall. The findings in this study confirm that hemicellulose is the main enrichment site for Cd in ramie. It also provides a theoretical basis for Cd enrichment breeding in ramie.