For small aspherical molds, it is difficult for the existing polishing method to take into account the correction of the surface error and the control of the uniformity of the surface roughness (SR) distribution, because the polishing tool is always larger than the small mold. Therefore, we used viscoelastic polyester fiber cloth to wrap the small steel ball as a polishing tool to adapt to the surface shape change of the aspherical mold, and designed a semi-flexible small polishing disc tool with microstructure, which can better adapt to the curvature change of aspherical surface and obtain better SR Ra. At the same time, a combined polishing method of constant speed and variable speed for screw feed was proposed to improve the uniformity of SR distribution in the paper. Then, a series of theoretical analysis and experimental verification were carried out in this paper to predict the tool influence function (TIF) of the two polishing tools and the effectiveness of the combined polishing method. In the experiment, a TIF bandwidth of about 0.46 mm was obtained with a small spherical polishing tool, which favors the surface shape correction of the small aspherical mold. The experiment of uniform removal with a small polishing disc tool was carried out to quickly reduce the Ra. Finally, the surface quality of the aspherical mold was effectively improved, combined with the constant speed and variable speed polishing modes of screw feed of the small spherical polishing tool and the smoothing effect of the small polishing disc tool. The peak valley (PV) of two small aspherical molds with an optical effective diameter less than 13 mm converged from 0.3572 μm and 0.2075 μm to 0.1282 μm and 0.071 μm, respectively. At the same time, the SR dispersion coefficient was reduced from 27.9% and 41.6% to 14.2% and 12.7%, respectively. The study provides a good solution for the surface quality control of small aspherical molds.