The Internet of Vehicles (IoV) counts for much in advancing intelligent transportation by connecting people, vehicles, infrastructures, and cloud servers (CS). However, the open-access wireless channels within the IoV are susceptible to malicious attacks. Therefore, an authentication key agreement protocol becomes essential to ensure secure vehicular communications and protect vehicle privacy. Nevertheless, although the vehicles in the group are compromised, they can still update the group key and obtain the communication content in the existing group key agreement protocols. Therefore, it is still challenging to guarantee post-compromise forward security (PCFS). Dynamic key rotation is a common approach to realizing PCFS, which brings a heavy computation and communication burden. To address these issues, an efficient and robust continuous group key agreement (ER-CGKA) scheme with PCFS is designed for IoV. The propose-and-commit flow is employed to support asynchronous group key updates. Besides, the computation cost and communication overhead are significantly reduced based on the TreeKEM architecture. Furthermore, we adopt the threshold mechanism to resist the collusion attacks of malicious vehicles, which enhances the ER-CGKA scheme’s robustness. Security analysis indicates that the proposed scheme satisfies all the fundamental security requirements of the IoV and achieves PCFS. The performance evaluation results show that our ER-CGKA scheme demonstrates a reduction in the computation cost of 18.82% (Client) and 33.18% (CS) approximately, and an increase in communication overhead of around 55.57% since pseudonyms are utilized to achieve conditional privacy-preserving. Therefore, our ER-CGKA scheme is secure and practical.