Urban air mobility (UAM) is a future air transportation system to solve the air pollution and movement efficiency problems of the traditional mobility system. In UAM environments, unmanned aerial vehicles (UAV) are used to transport passengers and goods providing various convenient services such as package delivery, air bus, and air taxi. However, UAVs communicate with ground infrastructures through open channels that can be exposed to various security attacks. Therefore, a secure mutual authentication scheme is necessary for UAM environments. Moreover, a handover authentication is also necessary to ensure seamless communication when the service location is changed. In this paper, we design a secure and efficient handover authentication scheme for UAM environments considering various security vulnerabilities and efficiency using elliptic curve cryptography (ECC). We utilize informal security analysis, Real-or-Random (RoR), Burrows-Abadi-Needham (BAN) logic, and Automated Validation of Internet Security Protocols and Applications (AVISPA) to prove the security of the proposed scheme. Furthermore, we compare the computation and communication cost comparisons of the proposed scheme with the other related schemes. The results show that the proposed scheme is secure and efficient for UAM environments.