Plastic anisotropy of Al alloy sheet is mainly caused by crystallographic texture, but it may be modulated by other microstructural parameters, including grain shape, arrangement of second phase particles and substructure, e.g. dislocation arrangement. In the present study, we have determined the anisotropic properties of thin sheet of the alloy AA 5182 in the cold rolled and in two back annealed (recovered) states. The texture combined with information on anisotropic grain morphology was utilised to model in-plane anisotropy with the help of polycrystal–plasticity simulations. It turned out that the aspect ratio of the substructural features in the as deformed microstructure, which was found to change significantly with progressing recovery, affects anisotropic behaviour besides the ubiquitous influence of crystallographic texture.