The open circuit voltage of lithium ion batteries in equilibrium state, as a vital thermodynamic characteristic parameter, is extensively studied for battery state estimation and management. However, the time-consuming relaxation process, usually for several hours or more, seriously hinders the widespread application of open circuit voltage. In this paper, a novel voltage relaxation model is proposed to predict the final open circuit voltage when the lithium ion batteries are in equilibrium state with a small amount of sample data in the first few minutes, based on the concentration polarization theory. The Nernst equation is introduced to describe the evolution of relaxation voltage. The accuracy and effectiveness of the model are verified using experimental data on lithium ion batteries with different kinds of electrodes (LiCoO2/mesocarbon-microbead and LiFePO4/graphite) under different working conditions. The validation results show that the presented model can fit the experimental results very well and the predicted values are quite accurate by taking only 5 min or less. The satisfying results suggest that the introduction of concentration polarization theory might provide researchers an alternative model form to establish voltage relaxation models.