As one of the most promising applications in future Internet of Things, Internet of Vehicles (IoV) has been acknowledged as a fundamental technology for developing the Intelligent Transportation Systems in smart cities. With the emergence of the sixth generation (6G) communications technologies, massive network infrastructures will be densely deployed and the number of network nodes will increase exponentially, leading to extremely high energy consumption. There has been an upsurge of interest to develop the green IoV towards sustainable vehicular communication and networking in the 6G era. However, as a special mobile ad-hoc network, the energy cost in an IoV system involves the communication and computation energy in addition to the fuel consumption and the electricity cost of moving vehicles. Moreover, the energy harvesting technology, which is likely to be adopted widely in 6G systems, will complicate the optimization of energy efficiency in the entire system. Current studies focus only on part of the energy issues in IoV systems without a comprehensive discussion of the state-of-the-art energy-efficient approaches and the influence of the development of 6G networks on green IoV. In this paper, we present the main considerations for green IoV from five different scenarios, including the communication, computation, traffic, Electric Vehicles (EVs), and energy harvesting management. The literatures relevant to each of the scenarios are compared from the perspective of energy optimization (e.g., with respect to resource allocation, workload scheduling, routing design, traffic control, charging management, energy harvesting and sharing, etc.) and the related factors affecting energy efficiency (e.g., resource limitation, channel state, network topology, traffic condition, etc.). In addition, we introduce the potential challenges and the emerging technologies in 6G for developing green IoV systems. Finally, we discuss the research trends in designing energy-efficient IoV systems.