An inerter system can amplify the deformation of its internal energy dissipation device, thereby improving the efficiency of energy dissipation and shock absorption. This is the so-called damping enhancement mechanism, one of the key mechanisms of the inerter system. Although the theoretical framework for damping enhancement of inerter systems has been established, the implementation of this principle for the design of an inerter system requires solving a complicated constrained optimization problem, which is not easy to be figured out using traditional approaches. To obtain valid design results through a lucid and robust method, it is proposed to optimize the damping parameters through a metaheuristic algorithm named harmony search algorithm in order to maximize the damping enhancement degree of the inerter system with the satisfaction of structural performance. First, the closed-form seismic response solutions of a single-degree-of-freedom (SDOF) structure with an inerter system are derived based on the theory of random vibration. Then, the mathematical expression of the constrained optimization problem is established. Due to the inefficiency of the original harmony search algorithm to solve the constrained optimization problem, the algorithm is modified by introducing a new harmony generating method and an adaptive strategy for parameter adjustment. The modified harmony search algorithm is compiled to solve the optimal design problem of the inerter system. The algorithm is verified by designing a structure with an inerter system. It is found that the number of iterations and time consumption until convergence required by the modified harmony search algorithm can be reduced by about 20%∼90% compared with the original algorithm, which confirms the effectiveness of the modified algorithm. The results of dynamic analyses show that the structure have achieved the preset performance demands under different cases and the damping enhancement characteristic of the inerter system is fully utilized.