The vibration caused by resonance modes frequently occurs during acceleration and deceleration of the modular joint integrated with flexible harmonic drive. The conventional equivalent rigid-body velocity method with observer can suppress the residual vibration induced by resonant frequency but has poor robustness to model uncertainties and external disturbances. Moreover, it cannot eliminate the torque ripple caused by the harmonic drive during low-speed uniform motion, reducing the velocity tracking accuracy. Hence, a velocity controller with a rigid-body state observer and an adjustable damper is designed to improve the robust performance and velocity tracking accuracy. The designed rigid-body state observer allows a higher gain so that the bandwidth of the observer can increase, and the equivalent rigid-body velocity can be acquired more accurately. Notably, the high gain observer reduces the sensitivity to model uncertainties and exotic disturbances, especially near the resonant frequency. In addition, the observer combined with an adjustable damper can suppress the residual vibration and torque ripple simultaneously. The proposed method is compared experimentally with a PI method and two other rigid-body velocity methods, such as the conventional equivalent rigid-body observer method and the self-resonance cancellation method, to verify its advantages.