This paper will solve one of the fractional mathematical physics models, a one-dimensional time-fractional differential equation, by utilizing the second-order quarter-sweep finite-difference scheme and the preconditioned accelerated over-relaxation method. The proposed numerical method offers an efficient solution to the time-fractional differential equation by applying the computational complexity reduction approach by the quarter-sweep technique. The finite-difference approximation equation will be formulated based on the Caputo’s time-fractional derivative and quarter-sweep central difference in space. The developed approximation equation generates a linear system on a large scale and has sparse coefficients. With the quarter-sweep technique and the preconditioned iterative method, computing the time-fractional differential equation solutions can be more efficient in terms of the number of iterations and computation time. The quarter-sweep computes a quarter of the total mesh points using the preconditioned iterative method while maintaining the solutions’ accuracy. A numerical example will demonstrate the efficiency of the proposed quarter-sweep preconditioned accelerated over-relaxation method against the half-sweep preconditioned accelerated over-relaxation, and the full-sweep preconditioned accelerated over-relaxation methods. The numerical finding showed that the quarter-sweep finite difference scheme and preconditioned accelerated over-relaxation method can serve as an efficient numerical method to solve fractional differential equations.