With the fast growth of the Industrial Internet of Everything (IIoE), computing and telecommunication industries all over the world are moving rapidly towards the IPv6 address architecture, which supports virtualization architectures such as Network Function Virtualization (NFV). NFV provides networking services like routing, security, storage, etc., through software-based virtual machines. As a result, NFV reduces equipment costs. Due to the increase in applications on Industrial Internet of Things (IoT)-based networks, security threats have also increased. The communication links between people and people or from one machine to another machine are insecure. Usually, critical data are exchanged over the IoE, so authentication and confidentiality are significant concerns. Asymmetric key cryptosystems increase computation and communication overheads. This paper proposes a lightweight and certificateless end-to-end secure communication scheme to provide security services against replay attacks, man-in-the-middle (MITM) attacks, and impersonation attacks with low computation and communication overheads. The system is implemented on Linux-based Lubuntu 20.04 virtual machines using Java programming connected to NFV-based large-scale hybrid IPv4-IPv6 virtual networks. Finally, we compare the performance of our proposed security scheme with existing schemes based on the computation and communication costs. In addition, we measure and analyze the performance of our proposed secure communication scheme over NFV-based virtualized networks with regard to several parameters like end-to-end delay and packet loss. The results of our comparison with existing security schemes show that our proposed security scheme reduces the computation cost by 38.87% and the communication cost by 26.08%.