We report a comparative polarized Raman study of Weyl semimetals TaAs, NbAs, TaP and NbP. The evolution of the phonon frequencies with the sample composition allows us to determine experimentally which atoms are mainly involved for each vibration mode. Our results confirm previous first-principles calculations indicating that the A1, B1(2), E(2) and E(3) modes involve mainly the As(P) atoms, the B1(1) mode is mainly related to Ta(Nb) atoms, and the E(1) mode involves both kinds of atoms. By comparing the energy of the different modes, we establish that the B1(1), B1(2), E(2) and E(3) become harder with increasing chemical pressure. This behaviour differs from our observation on the A1 mode, which decreases in energy, in contrast to its behaviour under external pressure.