The presence of strong ambient vibrations could have a negative impact on applications such as high precision inertial navigation and tilt measurement due to the vibration rectification error (VRE) of the accelerometer. In this paper, we investigate the origins of the VRE using a self-developed MEMS accelerometer equipped with an area-variation-based capacitive displacement transducer. Our findings indicate that the second-order nonlinearity coefficient is dependent on the frequency but the VRE remains constant when the displacement amplitude of the excitation is maintained at a constant level. This frequency dependence of nonlinearity is a result of several factors coupling with each other during signal conversion from acceleration to electrical output signal. These factors include the amplification of the proof mass’s amplitude as the excitation frequency approaches resonance, the nonlinearity in capacitance-displacement conversion at larger displacements caused by the fringing effect, and the offset of the mechanical suspension’s equilibrium point from the null position of the differential capacitance electrodes. Through displacement transducer and damping optimization, the second-order nonlinearity coefficient is greatly reduced from mg/g2 to μg/g2.