Inertial navigation system (INS) is widely used as inclinometer in coal drilling rig to realize navigation and attitude measurement. Vibration of coal drilling rig is one of the significant issues impacting the performance of alignment. Previous work has indicated that part of alignment error caused by vibration of gyroscope can be suppressed by inertial alignment frame. However, vibration error of accelerometer cannot be well suppressed. In this paper, an inertial alignment based on improved empirical mode decomposition (EMD) filter is proposed to address vibration issue in alignment. Moreover, a prediction strategy of extreme endpoints based on relativity of extreme points is proposed to reduce the endpoint error to suppress the accumulative error in EMD filter. The proposed alignment method is applied in laboratory condition. The result shows that the stability of accelerometer improves 42.43% at most, the alignment time reduce from 300 to 100 s and the alignment error of inclination angle is less than 5.4 arcmin. The experiment in coal mining rig shows the alignment time reduces from 300 to 50 s. The proposed alignment method improves the performance of inclinometer effectively and feasibly in alignment period and the improved EMD filter can be also applied in navigation period to suppress the vibration error of accelerometer.