Special concentrically braced frames (SCBFs) with in‐plane buckling (IPB) braces are often designed using the provisions applicable for out‐of‐plane buckling (OOPB) brace systems. As a result, the IPB brace systems may fail in the undesirable modes as observed in past experimental studies. In this study, a novel simplified analytical formulation based on the rigid beam spring model (RBSM) has been proposed to prevent the out‐of‐plane buckling of IPB braces. A high‐fidelity finite element model (FEM) is generated and validated with the experimental studies incorporating the influence of the residual stress, imperfection, and strain hardening. The variability, sensitivity, and uncertainties involved in predicting the direction of buckling of braces are examined. A parametric study has been carried out to investigate the influence of thickness and clearance of the gusset and knife plates, geometric imperfections, slenderness ratios, compactness ratios, residual stresses, and frame actions considering the most destabilizing effect in the OOPB mode. The third mode of buckling, termed as the mixed‐mode of buckling, has been observed in this study. A design criterion is proposed to get the desired IPB mode of buckling of braces in SCBFs. Multivariate regression analysis is carried out to incorporate the impact of various parameters as well as uncertainties in the formulation of the design criteria. The validity of the proposed equation is verified by comparing the findings of past experimental studies for various loading scenarios. Finally, a resistance factor of 0.8 in the LRFD approach has been proposed for the IPB brace systems.