The futuristic 5G network is designed to accommodate the increasing number of connected devices with higher speed, lower latency, and better security. To achieve these goals, two major strategies have been proposed recently: First, building the cellular network with many tiny base stations, 5G reduces the wireless transmission delays and improves spectral efficacy. Second, by allowing device-to-device (D2D) communication in the cellular network, it offloads the heavy network traffic among mobile devices in a distributed manner. However, due to the wireless nature of 5G communication, there will be many security challenges such as mutual authentication, device anonymity, end-to-end data secrecy, free-riding attack prevention, and so on. Although the Third Generation Partnership Project (3GPP) has recently standardized the 5G-AKA protocol for mutually authenticating user equipment (UE), base stations, and the core network, we found that there are still some security issues yet to resolve in the protocol. Furthermore, 3GPP has not standardized the authentication protocol for 5G-D2D communication, making D2D communication remain vulnerable in the 5G networks. In view of these issues, there is a need to develop authentication protocols that address all these security requirements in the 5G and 5G-D2D networks. The designed protocols should be computationally lightweight, such that all mobile devices including low-end smartphones and 5G-IoT connected devices can also enjoy 5G and 5G-D2D safely.To fill these research gaps, in this thesis, we first provide the background and a comprehensive literature survey to the security in 3GPP 5G and 5G-D2D network. We list out the security architecture of the 5G Core network and 5G D2D network, define their security requirements based on the current architecture, and then discuss the existing solutions by evaluating both their advantages and shortcomings. Next, we propose two security schemes, namely the Secure Blockchain-based Authentication and Key Agreement Scheme (5GSBA), and the Lightweight Traceable D2D authentication and Key Agreement Scheme (LT-AKA). 5GSBA aims to improve security in 5G, and LT-AKA aims to improve 5G-D2D.After that, for each proposal, we evaluate their security and logical correctness using formal verification tools. Performance evaluations for each scheme also show that the proposed schemes are computationally and spatially efficient while maintaining a high level of security.