Flatness plays an important role in the assembly process of aircraft engine rotors, especially the flange plane that is annular between the rotor shaft and cone. It determines the contact stiffness and assembly accuracy of the engine rotor. However, a specific method to measure the annular plane is lacking, especially in the on-machine conditions and for on-machine measurement. As a low-cost, high-precision and easy-to-use measurement method, error separating techniques (ESTs) are widely used for flatness measurement. However, when applying them, the initial error and probe fixture tilt error cannot be eliminated at the same time. This paper proposes a novel design of an on-machine measurement system and method for annular plane flatness, which can be applied to the assembly of aeroengine rotors. The novel method proposed in this paper combines the advantages of ESTs and utilizes the properties of the annular plane to eliminate the two main errors successfully. At the same time as processing data, the equivalent homogenization processing and 3D least squares method are introduced to further improve the credibility of the data. According to the method above, in order to meet the needs of on-machine measurement, a novel device that can adjust two axes and that is equipped with an indexing plate is designed. The measuring system uses the Keyence GT2-H12K contact probe, which can level itself by using the probe values to control two axes. Finally, the annular plane was measured by a three-coordinate measuring machine (CMM). The result of the CMM is 47.6 µm, which is close to the 45.9 µm measured by the novel on-machine conditions method, which proves the reliability of the method proposed in this paper.