Coal gangue (CG) represents a huge amount of industrial solid waste in China, and usually is used as a coarse aggregate to produce low-strength coal-gangue-based concrete. In this paper, in order to prove the possibility to obtain a higher-strength concrete with a higher CG utilization rate, reactive powder concrete (RPC) with coal gangue as a sand replacement at different replacement ratios was studied. RPC samples were prepared by replacing natural river sand (RS) with CG sand at different CG/RS weight ratios from 0–100% at intervals of 25%. Mechanical tests were carried out, and the microstructure features of RPC samples at 28 days were characterized. The test results showed that strong back shrinkage of strength existed. On days 7 and 14, the flexural strengths of samples with CG/RS replacement ratios of 0–75% fluctuated around the mean value. Strengths of samples with a CG/RS replacement ratio of 100% dropped off. However, on day 28, the flexural strengths of samples with CG were all lower than the strengths of samples on days 7 and 14. The flexural strengths and compressive strengths of the RPC with a CG/RS replacement ratio of 100% on day 14 were 14.09 MPa and 37.03 MPa, respectively, which decreased to 6.42 MPa and 28.44 MPa, respectively, on day 28. Compared with natural river sand, CG sand reduced the working performance, compressive strength, and flexural strength of RPC. Microscopic analysis showed that on day 28, increasing the CG replacement ratio could inhibit cement hydration, weaken the interface transition zone, and lead to the degradation of the RPC’s performance. Modification of CG sand would be helpful to obtain higher-strength concrete.