Sintering-free lightweight aggregates were prepared with high proportions of red mud and a binder material derived from whole solid wastes through rolling granulation at room temperature. The preparation process was optimized by changing the material matching and size parameters of the SFLAs. The physico-chemical performance, including the density, mechanical strength, water absorption, hydration products, heavy metal leaching, and microstructure were evaluated by jointly employing X-ray Fluorescence, X-ray Diffraction, and Inductively Coupled Plasma Optical Emission Spectrometry, Shadow Electron Microscope, etc. The results indicated that the red mud and waste-based binders were highly compatible in the granulation process, with up to 80% red mud being successfully added. The sintering-free lightweight aggregates products at the binder content of 30% and the size coverage of 10–16 mm exhibited a bulk density of 900–1000 kg·m−3, a 28 d cylinder compressive strength of 9.2–11.3 MPa, and water absorption of less than 10%. Owing to the formation of important hydration products, ettringite, the heavy metal leaching of the sintering-free lightweight aggregates was also proven to be environmentally acceptable. This work provides a promising pathway to prepare low-cost, high-strength, and green lightweight aggregates through the large-scale utilization of solid waste red mud.