Mobile core network uses Signaling System No. 7 (SS7) as its signaling system. SS7 takes charge of call setup, roaming, teardown messages, database queries and so on, and becomes the important goals of the attackers or the stealers as there is a large number of user's information, such as identity, location, and service, contained in signaling messages. The use of MTPsec at MTP3 layer in SS7 protocol stack is a good solution to provide secure protection for signaling messages in 3G core network. MTPsec consists of key exchange (KE) protocol and authentication header (AH) protocol. However, some leaks in KE protocol make the core network face serious threats. In this paper, we firstly discuss the mechanism of MTPsec, analyze the security of KE protocol, and then point out that the flaw in KE may cause "man-in-middle attack". Secondly, we propose a possible modification to prevent the MTPsec protocol from the attacks. Finally, we use BAN logic to make formal analysis on the security of the original and the modified authentication protocols in KE, respectively. It is shown that the modified protocol can offer the secure authentications between the initiator and the responder.