We consider one dimensional quantum circuits of the brickwork type, where the fundamental quantum gate is dual unitary. Such models are solvable: the dynamical correlation functions of the infinite temperature ensemble can be computed exactly. We review various existing constructions for dual unitary gates and we supplement them with new ideas in a number of cases. We discuss connections with various topics in physics and mathematics, including quantum information theory, tensor networks for the AdS/CFT correspondence (holographic error correcting codes), classical combinatorial designs (orthogonal Latin squares), planar algebras, and Yang-Baxter maps. Afterwards we consider the ergodicity properties of a special class of dual unitary models, where the local gate is a permutation matrix. We find an unexpected phenomenon: non-ergodic behaviour can manifest itself in multi-site correlations, even in those cases when the one-site correlation functions are fully chaotic (completely thermalizing). We also discuss the circuits built out of perfect tensors. They appear locally as the most chaotic and most scrambling circuits, nevertheless they can show global signs of non-ergodicity: if the perfect tensor is constructed from a linear map over finite fields, then the resulting circuit can show exact quantum revivals at unexpectedly short times. A brief mathematical treatment of the recurrence time in such models is presented in the Appendix by Roland Bacher and Denis Serre.