We present the results of GaAs annealing experiments in the conditions near equilibrium, which clarify the reasons of the transition from surface smoothing to roughening at temperatures above 650°C. The roughening is due to kinetic instabilities arising under deviation of annealing conditions towards growth or sublimation. These instabilities reveal themselves in appearing islands (for sublimation) and pits (for growth) of multilayer heights and depths, respectively. The islands and pits appear due to the motion of atomic steps through surface spots, at which sublimation and growth are suppressed. Pinning of the steps at these spots also lead to step bunching at surfaces with sufficiently small terrace widths. This explanation is consistent with Monte Carlo simulations of atomic processes on the GaAs surface. The similarity and distinctions in surface roughening under sublimation and growth, along with the role of Schwöbel barrier, are discussed. Annealing experiments in the cavities, which are formed by GaAs substrates with well-controlled atomic step densities enabled us to clarify the roughening mechanisms and to improve the efficiency of GaAs thermal smoothing technique by increasing smoothing temperature up to 775°C.