This work reported a novel magnetorheological elastomer (MRE) with different concentrations and sizes of abrasive grains, and its mechanical properties were investigated with or without magnetic field. Firstly, the MRE polishing wheel was prepared by molding with magnetic field, the microstructures of these MRE polishing wheels were observed, the magnetostrictive distribution of abrasive grains and ferromagnetic grains were discussed, and the magnetic properties was tested. Then, the magnetic hardness testing, magnetic compression experiments, and friction performance testing were conducted to investigate its mechanical properties and friction properties. Finally, a polishing experiment was designed to study the polishing performance of MRE polishing wheels. The results showed that the magnetic hardness, Young's modulus, and friction reduction of MRE polishing wheels were increased by adding abrasive grains, and the surface quality of workpiece after polishing was improved.