Backfill material used as a heat-transfer medium in boreholes of ground heat exchangers (GHEs) has a great influence on heat-transfer efficiency. Abandoned waste material causing environmental pollution has become a key issue around the world. To make full use of solid waste, backfill material made of waste fly ash in combination with graphite of high thermal conductivity was proposed. First, the thermal properties of cement/fly ash blended with different mass ratio of graphite were tested through laboratory tests. Then, a numerical model was established, in which the accuracy was validated based on a field test. Finally, an investigation of the long-term performance (over a period of 90 days) for four boreholes backfilled with natural sand, cement/fly ash, and cement/fly ash combined with different proportions of graphite was conducted through this numerical model, and the heat-transfer rates under constant inlet temperature in four boreholes decreased from 13.31, 44.97, 45.95, and 46.73 W/m to 14.18, 14.96, 15.66, and 16.19 W/m after the 90-day operation. Considering the influence of groundwater seepage, the horizontal groundwater flow had a positive impact, improving the long-term heat-transfer performance. The heat-transfer rates of four testing boreholes decreased from 44.46, 46.38, 47.22, and 47.68 W/m to 21.18, 21.93, 22.62, and 23.13 W/m. However, long-term groundwater seepage in a vertical direction caused a sharp decrease in the heat-transfer rate, and the values after 90 days were 10.44, 10.62, 10.78, and 10.81 W/m, which were the lowest of all the working conditions. The feasibility of using fly ash blended with graphite as backfill material was further validated through a comprehensive perspective, including indoor laboratory, field testing, and numerical simulation, which has rarely been conducted in previous research.