We investigate the influence of vibronic coupling on a molecular dimer strongly coupled to a single cavity mode. In the framework of the Holstein-Tavis-Cummings model, the energy structure of the molecular dimer is analyzed by numerical exact diagonalization and perturbation theory. Under numerical exact diagonalization, we find that the degeneracy of lower polaritons vanishes in the presence of vibronic coupling. Under the second-order degenerate perturbation theory, the degeneracy breaking of lower polaritons can be associated with asymmetric indirect interactions mediated by the upper polaritons and the dark states. The consistency of the two approaches confirms the robustness of our simulations, indicating that the vibration-induced symmetry breaking should be experimentally observed.