Wireless Mobile Ad Hoc Network (MANET) is a network of mobile nodes that is self-contained. The unique characteristics of mobile ad hoc networks (MANETs), such as dynamic topology and an open wireless medium can expose MANETs to a variety of security vulnerabilities. This has an effect on the protection of data transmitted between network nodes. Due to self-configuration and maintenance capabilities, the MANET faces a range of security problems as it expands its technology. Furthermore, because of the highly complex and resource-constrained existences of MANETs, conventional security solutions for wired networks are ineffective and inefficient. As a result, an effective technique is needed to prevent abnormal nodes following the detection process while also improving QOS parameters. The researchers in this paper use an optimization technique to improve efficient data transmission with high protection in the MANET. The optimal route is selected using modified ant colony optimization (MACO). To improve the transmission security of the MANET, an Elliptic curve Diffie-Hellman (ECDH) can be used. The Elliptic curve Diffie-Hellman (ECDH) method improves its overall efficiency and security. The packet delivery ratio, overhead, end to end delay and throughput are used in the security-based research. Finally, the results show that using optimization techniques, the MANET can achieve a high transmission rate while also improving data protection.