An internet of vehicles allows intelligent automobiles to interchange messages with other cars, traffic management departments, and data analysis companies about vehicle identification, accident detection, and danger warnings. The implementation of these features requires Internet of Things system support. Smart cars are generally equipped with many (hundreds or even thousands of) sensors and microcomputers so that drivers gain more information about travel. The connection between the in-vehicle network and the Internet can be leveraged by the attackers in a malicious manner and thus increases the number of ways the in-vehicle network can now be targeted. Protecting increasingly intelligent vehicle systems becomes more difficult, especially because a network of many different devices makes the system more vulnerable than ever before. The paper assumes a generic threat model in which attackers can access the controller area network (CAN) bus via common access points (e.g., Bluetooth, OBD-II, Wi-Fi, physical access, and cellular communication, etc). A machine learning based simplified attention (SIMATT)-security control unit (SECCU) symmetry framework is proposed towards a novel and lightweight anomaly detecting mechanism for the in-vehicle network. For this framework, we propose two new models, SECCU and SIMATT, and obtain state-of-the-art anomaly detecting performance when fusing the former to the latter. Regardless of the training phase or the detection phase, we strive to minimize the computational cost and thereby obtain a lightweight anomaly detection method. In particular, the SECCU model has only one layer of 500 computing cells and the SIMATT model has been improved to reduce its computational costs. Through substantial experiment comparisons (with various classical algorithms, such as LSTM, GRU, GIDS, RNN, or their variations), it is demonstrated that the SIMATT-SECCU framework achieves an almost optimal accuracy and recall rate.