A detailed numerical study of the separating and reattaching flow over a square leading-edge plate is presented, examining the instability modes governing transition from two- to three-dimensional flow. Under the influence of background noise, experiments show that the transition scenario typically is incompletely described by either global stability analysis or the transient growth of dominant optimal perturbation modes. Instead two-dimensional transition effectively can be triggered by the convective Kelvin–Helmholtz (KH) shear-layer instability; although it may be possible that this could be described alternatively in terms of higher-order optimal perturbation modes. At least in some experiments, observed transition occurs by either: (i) KH vortices shedding downstream directly and then almost immediately undergoing three-dimensional transition or (ii) at higher Reynolds numbers, larger vortical structures are shed that are also three-dimensionally unstable. These two paths lead to distinctly different three-dimensional arrangements of vortical flow structures. This paper focuses on the mechanisms underlying these three-dimensional transitions. Floquet analysis of weakly periodically forced flow, mimicking the observed two-dimensional quasi-periodic base flow, indicates that the two-dimensional vortex rollers shed from the recirculation region become globally three-dimensionally unstable at a Reynolds number of approximately 380. This transition Reynolds number and the predicted wavelength and flow symmetries match well with those of the experiments. The instability appears to be elliptical in nature with the perturbation field mainly restricted to the cores of the shed rollers and showing the spatial vorticity distribution expected for that instability type. Indeed an estimate of the theoretical predicted wavelength is also a good match to the prediction from Floquet analysis and theoretical estimates indicate the growth rate is positive. Fully three-dimensional simulations are also undertaken to explore the nonlinear development of the three-dimensional instability. These show the development of the characteristic upright hairpins observed in the experimental dye visualisations. The three-dimensional instability that manifests at lower Reynolds numbers is shown to be consistent with an elliptic instability of the KH shear-layer vortices in both symmetry and spanwise wavelength.