In this paper, cables are proposed to connect the inerter and main frame for translation-to-rotation conversion, i.e., the cable-bracing inerter system (CBIS), with a magnified mass and enhanced damping effect. This novel configuration has the benefits of deformation relaxation at the connecting joints, easy installation, and an adaptive layout for nonconsecutive-story deployment. Dynamic motion equations were established for a single degree-of-freedom (SDOF) model equipped with a CBIS. The influence of dimensionless parameters, such as inertance-mass ratio, stiffness ratio and additional damping ratio on vibration mitigation were studied in terms of displacement response and force output. A single objective and multiple objective optimal design method were developed for a CBIS-equipped structure based on a performance-oriented design framework. Finally, the mitigation effect was illustrated and verified by a numerical simulation in a time-domain. The results showed that a CBIS is an effective structural response mitigation device used to mitigate the response of structural systems under earthquake excitation. Using the proposed optimization method, CBIS parameters can be effectively designed to satisfy the target vibration control level.