Access control systems are the most utilized preventive and protective solution for guaranteeing highly secure and monitored environments where people can move about and live in complete safety. The Smart Building Access Management System using Digital Keys provides an efficient means of granting or revoking access to users in residential and commercial buildings. It responds to the ever-increasing demand for internet-connected devices and the need for a system that is secure, convenient, and easy to manage. Despite the pivotal role of access control systems, the current technological standing shows severe security vulnerabilities, a lack of practical management solutions and a non-optimized user conveniency. Majorly used credential technologies show an absence of encryption capabilities. Some user management solutions do not scale well and present a lack of proper scalability. Our proposed system in this paper is the Attribute-Based Access Control System (ABACS) for Smart Building Access Management System, which offers an internet-oriented physical access control system, based on an end-to-end secured solution, an easy-to-use hybrid cloud-based system for effective access management and a mobile user application for optimal convenience. Authentication, integrity, and confidentiality are guaranteed using multiple security methods, including a Trusted Execution Environment (TEE) for a safe digital key storage and encryption, and the Transport Layer Security (TLS) protocol for secured channel communication, supported by a trusted third-party Certification Authority (CA). The Near-Field Communication (NFC) channel is used for quick key sharing. Access policies and user management is achieved using the hybrid fog-cloud paradigm and the Attribute-Based Access Control (ABAC) model. Finally, user convenience and optimal user experience are reached by means of an aesthetic mobile application for digital key generation and storage. The testing results and performance evaluation show that our proposed system’s backend, efficiently processes requests in both sequential and concurrent scenarios. With sequential requests, our proposed system’s longest-performing request outperforms iPACS’s by at least tenfold. In addition, with concurrent requests, our system shows a performance at least twice better than iPACS’s.