Internet of Drones (IoD), designed to coordinate the access of unmanned aerial vehicles (UAVs), is a specific application of the Internet of Things (IoT). Drones are used to control airspace and offer services such as rescue, traffic surveillance, environmental monitoring, delivery and so on. However, IoD continues to suffer from privacy and security issues. Firstly, messages are transmitted over public channels in IoD environments, which compromises data security. Further, sensitive data can also be extracted from stolen mobile devices of remote users. Moreover, drones are susceptible to physical capture and manipulation by adversaries, which are called drone capture attacks. Thus, the development of a secure and lightweight authentication scheme is essential to overcoming these security vulnerabilities, even on resource-constrained drones. In 2021, Akram et al. proposed a secure and lightweight user–drone authentication scheme for drone networks. However, we discovered that Akram et al.’s scheme is susceptible to user and drone impersonation, verification table leakage, and denial of service (DoS) attacks. Furthermore, their scheme cannot provide perfect forward secrecy. To overcome the aforementioned security vulnerabilities, we propose a secure mutual authentication and key agreement scheme between user and drone pairs. The proposed scheme utilizes physical unclonable function (PUF) to give drones uniqueness and resistance against drone stolen attacks. Moreover, the proposed scheme uses a fuzzy extractor to utilize the biometrics of users as secret parameters. We analyze the security of the proposed scheme using informal security analysis, Burrows–Abadi–Needham (BAN) logic, a Real-or-Random (RoR) model, and Automated Verification of Internet Security Protocols and Applications (AVISPA) simulation. We also compared the security features and performance of the proposed scheme and the existing related schemes. Therefore, we demonstrate that the proposed scheme is suitable for IoD environments that can provide users with secure and convenient wireless communications.