The effect of curvature on the three-dimensional instabilities of lid-driven flow in a toroidal square cavity is numerically studied using the linear stability analysis based on the spectral element method. The critical stability curves are determined, and four types of instability modes are predicted. For small curvatures (δ ≤ 0.2), the most dangerous mode is a steady short-wavelength mode (λc ≈ 0.4), which is consistent with that in the straight square cavity. In a narrow range of 0.22 ≤ δ ≤ 0.28, the critical mode is oscillatory and of medium wavelength (0.84 < λc < 1.07). For δ ≥ 0.29, two types of steady long-wavelength modes (λc > 1.26) become the critical mode. The disturbance energy analyses show that all the instabilities are caused by the centrifugal instability mechanism.