Megatall and supertall buildings often adopt megastructure systems characterized by secondary structure systems, and the serviceability problem caused by wind-induced vibrations often becomes the dominant factor in the structural design. Because the deformation of a supertall building usually presents bending characteristics, a viscous damped outrigger can reduce the wind-induced vibration of a supertall building with the installation of a small number of viscous dampers. However, time history analysis of the prototype model considering the nonlinear characteristics of viscous dampers is time-consuming, which is not conducive for iterative design optimization. Additionally, the conventional simplified model composed of one cantilever beam cannot be used for the analysis and design of a viscous damped outrigger. In this study, a simplified wind-induced vibration prediction model is proposed based on the mechanical characteristics of megastructures. This simplified model is a plane model that includes both core walls and frames whose member size can be extracted from the original structure. Parametric analysis shows that the simplified model has high acceleration prediction accuracy. An optimal design method combined with the simplified model, which aims to minimize the damped outrigger system cost, is proposed. A 600-m supertall building is presented as a case study. The accuracy and effectiveness of the simplified model and the optimal design method proposed in this study are illustrated. Thus, applying this optimal design method in combination with the simplified model can save significant analysis and design time and is conducive to the application of viscous damped outriggers in practical engineering.