This study evaluated the potential and environmental benefits of using oil shale residue as a replacement for fine aggregate in concrete through a series of experiments. Initially, the crushing value test confirmed the oil shale residue’s value at 16.7%, meeting the load-bearing standards for fine aggregates, thus proving its viability as a complete substitute. Further, the oil shale residue was treated with a 60 mg/L concentration of Tween 80 and other surfactants for oil removal. The treated concrete specimens demonstrated excellent compressive performance and a dense internal structure. Building on this, the mechanical properties of the oil shale residue concrete were explored across different replacement ratios (from 40% to 100%), revealing an increase in compressive strength with higher replacement ratios. In the durability tests, compared to the JZ group, the oil shale residue concrete modified with desulfurization gypsum exhibited a 0.03% reduction in mass loss rate and a 10.13% reduction in relative moving elasticity modulus loss rate, particularly noticeable after 175 freeze–thaw cycles where specimens B1 to B4 exhibited no significant damage, highlighting its remarkable durability. Overall analysis indicated that using oil-removed oil shale residue as a substitute for fine aggregate in concrete, combined with desulfurization gypsum modification, not only enhances concrete performance but also significantly reduces the consumption of natural aggregates and environmental pollution, promoting resource utilization and sustainable development.