Green roadside units (RSUs), also called renewable energy-powered RSUs, are utilized recently rather than the traditional electric-powered RSUs with high power consumption and the large infrastructure deployment cost in the Internet of vehicles (IoVs). However, the power of the green RSUs is limited and unstable, which is affected by the battery size and charging environment. Therefore, a big challenge to deploy green RSUs in the IoVs is to schedule their service process properly, in order to extend the service efficiency of RSUs. In this paper, a deep learning-based communication scheduling mechanism is proposed regarding the service scheduling problem. In particular, a three-part scheduling algorithm consisting of RSU clustering, deep learning-based traffic prediction, and a vehicle access scheduling algorithm is presented to maximize the service number of vehicles and minimize the energy cost. An extensive simulation is done, and the simulation results indicate that our algorithm can serve more vehicles with less energy consumption compared with other scheduling mechanisms under different scenarios.