The blockchain-empowered Internet of Vehicles (IoV) enables various services and achieves data security and privacy, significantly advancing modern vehicle systems. However, the increased frequency of data transmission and complex network connections among nodes also make them more susceptible to adversarial attacks. As a result, an efficient intrusion detection system (IDS) becomes crucial for securing the IoV environment. Existing IDSs based on convolutional neural networks (CNN) often suffer from high training time and storage requirements. In this paper, we propose a lightweight IDS solution to protect IoV against both intra-vehicle and external threats. Our approach achieves superior performance, as demonstrated by key metrics such as accuracy and precision. Specifically, our method achieves accuracy rates ranging from 99.08% to 100% on the Car-Hacking dataset, with a remarkably short training time.