There are some good combinatorial structures suitable for image encryption. In this study, a new chaotic image encryption algorithm based on transversals in a Latin square is proposed. An n -transversal has the function of classifying all the matrix's positions, and it can provide a pair of orthogonal Latin squares. Employing an n -transversal of a Latin square, we can permutate an image data group by group for the first time, then use two Latin squares for auxiliary diffusion based on a chaotic sequence, finally make use of a pair of orthogonal Latin squares to do the second scrambling. As a whole, the encryption process of “scrambling-diffusion-scrambling” is formed. The experimental results indicate that this algorithm achieves a secure and fast encryption effect. The final information entropy is very close to 8, and the correlation coefficient is approximate to 0. All these tests verify the robustness and practicability of this proposed algorithm.