Frictionally induced vibrations in rubber are readily triggered due to their lower stiffness and higher elasticity. This study developed a numerical model to investigate the frictional vibration of a rubber block with a groove on its side surface against an aluminum disc. The results indicate that a backside groove (GB) on the block significantly enhances vibration attenuation, with a decay time 0.6 s faster than a non-grooved (NG) block, despite a potentially higher initial vibrational amplitude. In contrast, a frontside groove (GF) results in persistent frictional oscillations, with the steady-state time being similar for both GB and GF configurations. The underlying mechanism is attributed to the GB’s effectiveness in reducing the maximum energy imparted to the block initially, dissipating vibrational energy more swiftly, and distributing the contact stress more uniformly. The discrepancies in frictional forces between the conducted experiment and the simulation for the NG, GB and GF cases were 11.3%, 9.3% and 12.1%, respectively, quantitatively indicating the moderate precision of the results from the simulation. The insights gained from this study hold promise for enriching methods of mitigating vibrations arising from rubber friction.