The clearance between the core plate and restraining part of the buckling‐restrained brace using steel mortar planks (BRBSM) considerably affects its failure mode status and hysteretic characteristics. When the clearance is controlled by average, in some cases, the clearance exceeds its average value by local unevenness on infilled material surface. As a continuation of our study on the BRBSM with controlled clearance, effects of the unevenness on the infilled material surface of the restraining parts on the structural performance of the buckling‐restrained brace are examined and the possibility of securing a clearance that allows for the evenness is shown. A cyclic axial loading test is conducted using specimens with the evenness on infilled material surface; the effects on the failure mode, strength increase ratio, compression‐to‐tension strength ratio, and cumulative plastic strain energy ratio of the BRBSM are evaluated. Furthermore, the propagation process of the higher‐order buckling modes and the unevenness on infilled material surface are examined, measuring the strain distribution of the core plate under compressive and tensile loading.