Internet of Vehicles (IoV) is treated as an extension of Vehicle-to-Vehicle (V2V) communication network. IoV helps in enhancing driving aids with the help of vehicle Artificial Intelligence (AI) awareness of other vehicles and their actions. IoV is connected in an adhoc networking environment which utilizes each vehicle in the network as a node, called Vehicular Ad Hoc Network (VANET), where the vehicles may be also connected to the public Internet. It is specifically important for the autonomous vehicles because they can instantaneously communicate with other vehicles surrounding them. In addition, safely avoiding accident prone zones is crucial in order to continue secure and smart transportation. Since the communication among various entities involved in the IoV environment is via open channel, it gives an opportunity to a passive/active adversary to intercept, modify, delete or even insert fake information during communication. It is then a serious concern for the vehicles users to determine whether the received information is genuine. In this survey paper, various security aspects, threats and attacks, network and threat models related to the IoV environment are discussed. Next, a taxonomy of security protocols is given that is essential to provide IoV data security. In particular, focus on various authentication protocols is given that is needed for mutual authentication among the involved entities in the IoV environment for secure communication. A detailed comparative analysis among various state-of-art authentication protocols proposed in the related IoV environment is provided to show their effectiveness as well as security and functionality features. Moreover, some testbeds are described that were designed and implemented for the IoV environment. In addition, some future challenges for IoV security protocols are also highlighted that are necessary to address in the future. INDEX TERMS Internet of Vehicles (IoV), vehicular adhoc networks (VANETs), authentication, batch verification, security.