In order to study the effect of M substitution of transition elements on the structural properties of R2Co7−xMxB3 (R = Y, Gd; M = Ti, V, Cr) compounds, the phase stability and site preference are performed by using the pair potentials based on the lattice inversion technique. In R2Co7−xMxB3, M atoms substitute for Co with a strong preference for the 6i1 sites and the order of site preference is 6i1, 2c and 6i2. Calculated lattice parameters are in good agreement with the experimental data. Moreover, the total and partial phonon densities of states are first evaluated for the R2Co7B3 compounds with the hexagonal Ce2Co7B3-type structure.