Selective laser melting is receiving increasing interest as an additive manufacturing technique. Residual stresses induced by the large temperature gradients and inhomogeneous cooling process can favour the generation of cracks. In this work, a crystal plasticity finite element model is developed to simulate the formation of residual stresses and to understand the correlation between plastic deformation, grain orientation and residual stresses in the additive manufacturing process. The temperature profile and grain structure from thermal-fluid flow and grain growth simulations are implemented into the crystal plasticity model. An element elimination and reactivation method is proposed to model the melting and solidification and to reinitialize state variables, such as the plastic deformation, in the reactivated elements. The accuracy of this method is judged against previous method based on the stiffness degradation of liquid regions by comparing the plastic deformation as a function of time induced by thermal stresses. The method is used to investigate residual stresses parallel and perpendicular to the laser scan direction, and the correlation with the maximum Schmid factor of the grains along those directions. The magnitude of the residual stress can be predicted as a function of the depth, grain orientation and position with respect to the molten pool. The simulation results are directly comparable to X-ray diffraction experiments and stress–strain curves.