All-optical devices have a great potential in optical communication systems. As a new material, graphene has attracted great attention in the field of optics due to its unique properties. We propose a graphene-assisted polymer optically controlled thermo-optic switch, based on the Ex01 mode, which can reduce the absorption loss of graphene. Graphene absorbs 980 nm pump light, and uses the heat generated by ohmic heating to switch on and off the signal light at 1550 nm. The simulation results show that, when the graphene is in the right position, we can obtain the power consumption of 9.5 mW, the propagation loss of 0.01 dB/cm, and the switching time of 127 μs (rise)/125 μs (fall). The switching time can be improved to 106 μs (rise) and 102 μs (fall) with silicon substrate. Compared with an all-fiber switch, our model has lower power consumption and lower propagation loss. The proposed switch is suitable for optically controlled fields with low loss and full polarization. Due to the low cost and easy integration of polymer materials, the device will play an important role in the fields of all-optical signal processing and silicon-based hybrid integrated photonic devices.