In view of the increasing concerns over non-renewable resource depletion and waste management, this paper studied the development of low-density controlled low-strength material (CLSM) by using stone sludge and lightweight aggregates. First, the investigation was performed at a laboratory scale to assess the effects of the composition on the properties of the resulting low-density CLSM. The Taguchi method with an L9(34) orthogonal array and four controllable three-level factors (i.e., the stone sludge dosage, water to binder ratio, accelerator dosage and lightweight aggregate dosage) was adopted. Then, to optimize the selected parameters, the analysis of variance method was used to explore the effects of the experimental factors on the performance (fresh and hardened properties) of the produced low-density CLSM. The test results show that when the percentage of stone sludge usage was increased from 30% to 60%, the initial setting time approximately doubled on average. Moreover, at the age of 28 days, the compressive strength of most specimens did not exceed the upper limit of 8.83 MPa stipulated by Taiwan’s Public Construction Commission. Further, the material cost per cubic meter of the produced CLSM was about NT$ 720.9 lower than that of the ordinary CLSM, which could reduce the cost by 40.6%. These results indicate that the use of stone sludge as a raw material to produce CLSM could achieve environmental sustainability. In other words, the use of stone sludge and lightweight aggregates to produce low-density CLSM was extremely feasible.