Variable transmission ratio racks show great potential in rice transplanters as a key component of variable transmission ratio steering to balance steering portability and sensitivity. The objective of this study was to develop a novel geometrical design method to achieve quick, high-quality modeling of the free curvilinear tooth profile of a variable transmission ratio rack. First, a discrete envelope motion 3D model was established between the pinion-sector and the variable transmission ratio rack blank based on the mapping relationship between the rotation angle of the pinion-sector and the displacement of the rack, according to the variable transmission ratio function. Based on the loop Boolean subtraction operation, which removed the pinion-sector from the rack blank during all moments of the discrete motion process, the final complex changing tooth shape of the variable transmission ratio rack was enveloped. Then, since Boolean cutting residues made the variable ratio tooth surface fluctuant and eventually affected the precision of the model, this study proposed a modification method for establishing a smooth and continuous tooth profile. First, a novel fitting algorithm used approximate variable ratio tooth profile points extracted from the Boolean cutting marks and generated a series of variable ratio tooth profiles by utilizing B-spline with different orders. Next, based on a transmission stability simulation, the variable ratio tooth profile with optimal dynamic performance was selected as the final design. Finally, tests contrasting the transmission stability of the machining samples of the initial variable ratio tooth profile and the final variable ratio tooth profile were conducted. The results indicated that the final variable ratio tooth profile is more effective than the initial variable ratio tooth profile. Therefore, the proposed variable ratio tooth profile modeling and modification method for eliminating Boolean cutting residues and improving surface accuracy is proved to be feasible.