Set-back modification represents a dependable method to reducing the wind effects on high-rise buildings, but the flow mechanism and aerodynamic responses of set-back tall buildings have not been explored systematically. Using the square building as the benchmark model, six set-back building models with varying steps and sizes were chosen for numerical simulation in this study. The results of mean wind pressure and local wind force distribution, power spectral densities of base moment, flow field, and structural response of the proposed building models are compared and analysed using the Large Eddy Simulation (LES) technique. The results demonstrate that the aerodynamic performance of set-back tall buildings dramatically improved, with the actual optimization in the cross-wind direction being more significant. Generally, a single set-back measure can enhance the aerodynamic performance of high-rise buildings more effectively than a double set-back measure. The outcome of this study provides an aerodynamic design guide for set-back tall buildings.