Noniridescent and nonfading structural colors generated from metallic and dielectric nanoparticles with extraordinary optical properties hold great promise in applications such as image display, color printing, and information security. Yet, due to the strong wavelength dependence of optical constants and the radiation pattern, it is difficult and time-consuming to design nanoparticles with the desired hue, saturation, and brightness. Herein, we combined the Monte Carlo and Mie scattering simulations and a bidirectional neural network (BNN) to improve the design of gold nanoparticles’ structural colors. The optical simulations provided a dataset including color properties and geometric parameters of gold nanoparticle systems, while the BNN was proposed to accurately predict the structural colors of gold nanoparticle systems and inversely design the geometric parameters for the desired colors. Taking the human chromatic discrimination ability as a criterion, our proposed approach achieved a high accuracy of 99.83% on the predicted colors and 98.5% on the designed geometric parameters. This work provides a general method to accurately and efficiently design the structural colors of nanoparticle systems, which can be exploited in a variety of applications and contribute to the development of advanced optical materials.