Uncertainty in modal characteristics due to output-only system identification methods has been a challenge in operational modal analysis. The present study aims to extract modal parameters of Karun IV Dam (the highest arch dam in Iran) using the balanced stochastic subspace identification (B-SSI) and investigate the influence of user-defined parameters (i.e., columns and block rows of Hankel Matrix) on the uncertainty of the results. The effects of noise caused by numerical instabilities were first filtered using the inverse process by the condition number. Subsequently, the modal properties were homogenized with spatial clustering of applications with noise (DBSCAN) to remove the outlier and spurious characteristics. Then, the physical modes were validated by inspecting the complexity of the mode shapes based on the mode complexity factor criterion. Finally, the coefficient of variation (CV) of the validated clusters was employed to conduct a sensitivity analysis performed concerning the dimensions of the Hankel matrix to find the optimal models (with the minimum error in estimating the modal characteristics). The results indicated that the proposed method prevented the emergence of computational and noisy modes by regulating the extracted models, such that the first model of the structure was extracted with an error of less than 10% compared to the numerical model.