MAB phases are layered ternary compounds with alternative stacking of transition metal boride layers and group A element layers. Until now, most of the investigated MAB phases are concentrated on compounds with Al as the A element layers. In this work, the family of M5SiB2 (M = IVB—VIB transition metals) compounds with silicon as interlayers were investigated by density functional theory (DFT) methods as potential MAB phases for high-temperature applications. Starting from the known Mo5SiB2, the electronic structure, bonding characteristics, and mechanical behaviors were systematically investigated and discussed. Although the composition of M5SiB2 does not follow the general formula of experimentally reported (MB)2zAx(MB2)y (z = 1, 2; x = 1, 2; y = 0, 1, 2), their layered structure and anisotropic bonding characteristics are similar to other known MAB phases, which justifies their classification as new members of this material class. As a result of the higher bulk modulus and lower shear modulus, Mo5SiB2 has a Pugh’s ratio of 0.53, which is much lower than the common MAB phases. It was found that the stability and mechanical properties of M5SiB2 compounds depend on their valence electron concentrations (VECs), and an optimum VEC exists as the criteria for stability. The hypothesized Zr and Hf containing compounds, i.e., Zr5SiB2 and Hf5SiB2, which are more interesting in terms of high-temperature oxidation/ablation resistance, were found to be unfortunately unstable. To cope with this problem, a new stable solid solution (Zr0.6Mo0.4)5SiB2 was designed based on VEC tuning to demonstrate a promising approach for developing new MAB phases with desirable compositions.