The rotary vector reducer presents high precision and load capacity characteristics. The shape of the cycloidal gear tooth profile in the rotary vector reducer significantly affects its performance. Meanwhile, the effect of the machining error on the error between the designed and theoretical tooth profiles cannot be ignored. Thus, this paper analyzes the machining error items that affect the shape of the cycloidal gear profile from the machining process perspective. Due to the random characteristics and different distribution rules inherent in various machining errors, this paper proposes a cycloidal gear machining error compensation and modification model based on the Monte Carlo simulation method, providing a new theoretical method for compensating cycloidal gear machining errors. While compensating for machining errors, considering the impact of cycloidal gear modification on the carrying capacity, the NSGA-Ⅱoptimization algorithm is utilized to optimize the cycloidal gear modification parameter and finally, to solve the modification parameters with a more comprehensive performance.