Based on magnetometer and gyro measurement, a sequential scheme is proposed to determine the orbit and attitude of small satellite simultaneously. In order to reduce the impact of orbital errors on attitude estimation, a robust adaptive Kalman filter is developed. It uses a scale factor and an adaptive factor, which are constructed by Huber function and innovation sequence, respectively, to adjust the covariance matrix of system state and observational noise, change the weights of predicted and measured parameters, get suitable Kalman filter gain and approximate optimal filtering results. Numerical simulations are carried out and the proposed filter is approved to be robust for the noise disturbance and parameter uncertainty and can provide higher accuracy attitude estimation.