In this paper, a dwell time optimization method based on the particle swarm optimization algorithm is proposed according to the pulse iteration principle in order to achieve high-precision magnetorheological finishing of optical components. The dwell time optimization method explores the optimal solution in the solution space by comparing the accuracy value of the final surface with the set value. In this way, the dwell time optimization method was able to achieve global optimization of the overall dwell times and each dwell time point, ultimately realizing the high-precision processing of a surface. Through the simulation of two Φ156 mm asphaltic mirrors (1# and 2#), the root-mean-square (RMS) and peak–valley (PV) values of 1# converged from the initial values of 169.164 nm and 1161.69 nm to 24.79 nm and 911.53 nm. Similarly, the RMS and PV values of 2# converged from the initial values of 187.27 nm and 1694.05 nm to 31.76 nm and 1045.61 nm. The simulation results showed that compared with the general pulse iteration method, the proposed algorithm could obtain a more accurate dwell time distribution of each point under the condition of almost the same processing time, subsequently acquiring a better convergence surface and reducing mid-spatial error. Finally, the accuracy of the optimization algorithm was verified through experiments. The experimental results demonstrated that the optimized algorithm could be used to perform high-precision surface machining. Overall, this optimization method provides a solution for dwell time calculation in the process of the magnetorheological finishing of optical components.