Sensor noise, such as low-frequency drift and white noise, often causes large measurement errors when an oscillation method is used to determine rotational inertia of a rigid body. A demodulation algorithm based on differentiation and resonance frequency analysis of torsional oscillation signals is proposed in this paper to eliminate the influence of sensor noise on inertia measurement. By using a torsion pendulum method as an example, a measurement approach involving a kinetics differential equation of damped torsional oscillation movement is developed and, on this basis, a rotational inertia measurement function in relation to the amplitude modulation parameter and oscillation frequency is derived. In the proposed algorithm, oscillation detection signals are first differentiated to extract local extremum points and calculate the amplitude modulation parameter. Next, with a certain frequency step, a series of reference signals are constructed and mixed with the original signals. The maximum product sum is calculated to determine the resonance frequency that is exactly equal to the oscillation frequency. Finally, the measurand can be determined by the measurement function. The simulation and experimental results show that this algorithm can effectively reduce the influence of sensor noise on measurement. The comparison of measurement results with other signal processing methods and other measurement systems is carried out, which verifies better effectiveness of the proposed method.