The influences of high-frequency vibrations on the Rayleigh–Marangoni instability in a two-layer system are investigated theoretically in the framework of the averaging method. We focus on the effects of vertical and horizontal vibrations on the stability of different convection modes. The results show that vertical vibrations significantly stabilize the system, while horizontal vibrations significantly destabilize it. In the presence of vertical vibrations, instability only occurs in a system heated from below. However, in the presence of horizontal vibrations, instability can also occur in a system cooled from below. When Marangoni effect is dominant at the interface, it is found that there are four types of coupling modes. The oscillatory convection is the result of the competition between different modes. In the presence of Marangoni effect at the interface, the structure of the interfacial flow is complicated. In some cases, small counter-rolls may develop to preserve the nonslip condition of fluids in either the upper layer or the lower layer.