A specially designed cleaning system was adopted to wash away mud but keep coarse particles and stone powders in quarry waste. Then the cleaned quarry waste was used as concrete fine aggregate. Size distributions of all concrete constituents were analyzed, and workability and compressive strength of concrete were determined to evaluate gradation effect of cleaned quarry waste to improve size distributions of concrete constituents, thus improving concrete properties. Results showed that the designed cleaning system can wash away mud and keep stone powders and coarse particles in quarry waste. The particle sizes of stone powders were between 0.075 mm and 0.15 mm, and those of coarse particles were between 0.15 mm and 4.75 mm. When the cleaned quarry waste was used, continuous size distributions of all concrete constituents were formed, which solved discontinuous size distribution problems among cement, fine, and coarse aggregates. Therefore, concrete workability was improved and compressive strengths at different ages were increased.