This work presents a novel cluster based optimization procedure for estimating parameter values that yield stable, periodic responses with desired amplitude in nonlinear vibrating systems. The parameter values obtained by conventional nonlinear optimization schemes, with minimization of amplitude as the objective, may not furnish periodic and stable responses. Moreover, global optimization strategies may converge to isolated optima that are sensitive to parametric perturbations. In practical engineering systems, unstable or isolated optimal orbits are not practically realizable. To overcome these limitations, the proposed method tries to converge to a cluster in the r-dimensional parameter space in which the design specifications including the specified optimality, periodicity, stability and robustness are satisfied. Thus, it eliminates the need for computationally expensive bifurcation studies to locate stable, periodic parameter regimes before optimization. The present method is based on a hybrid scheme which involves the algebraic form of the governing equations in screening phase and its differential form in the selection phase. The method is validated with Den Hartog's vibration absorbers and is then applied to vibration absorbers with material nonlinearity and asymmetric vibration isolators with geometric nonlinearity. In both the cases, the converged cluster is shown to yield stable, periodic responses satisfying the amplitude condition. Parametric perturbation studies are conducted on the cluster to illustrate its robustness.