This paper presents a new method to determine the cutter-workpiece engagement (CWE) for a toroidal cutter during free-form surface machining in five-axis milling. A hybrid method, which is a combination of a discrete model and an analytical approach, was developed. Although the workpiece surface was discretized by a number of normal vectors, there was no calculation to determine the intersection between the normal vector and the cutting tool. The normal vectors were used to define the workpiece surface mathematically; next, the engagement point was calculated using a combination of the workpiece surface equation, the parametric equation of the cutting tool, and the tool orientation data. Three model parts with different surface profiles were tested to verify the validity of the proposed method; the results indicated that the method was accurate. The method also eliminated the need for a large number of discrete vectors to define the workpiece surface. A comparison showed that the proposed method was computationally more efficient. The CWE model was subsequently applied to support the cutting force prediction model. A validation test demonstrated that in terms of trends and amplitudes, the predicted cutting forces exhibit good agreement with the cutting force generated experimentally.