Analytical coarse alignment and Kalman filter fine alignment based on zero-velocity are typically used to obtain initial attitude for inertial navigation systems (SINS) on a static base. However, in the shipboard mooring state, the static observation condition is corrupted. This paper presents a rapid alignment method for SINS on swaying bases. The proposed method begins with a coarse alignment technique in the inertial frame to obtain an initial rough attitude. Subsequently, a Kalman filter with position updates is employed to estimate the remaining misalignment error. To enhance the filter estimation performance, an appropriate lower boundary is set to the target states’ variances according to a carefully designed relative convergence index. The variance-constraint Kalman filter (VCKF) approach is proposed in this paper, and the shipborne experiments validate its effectiveness. The results demonstrate that the VCKF approach significantly reduces the time requirement for fine alignment to achieve the same accuracy on a swaying base, from 90 min in the classic Kalman filter to 30 min. Additionally, the parameter estimation performance in the Kalman filter is also improved, particularly in situations where unpredicted external interference is involved during fine alignment.