Various IoT-based applications such as smart home, intelligent medical, and VANETs have been put into practical utilization. The smart home is one of the most concerned environments, allowing users to access and control smart devices via the public network remotely. The smart home can provide many intelligent services for users through these smart devices. To securely access devices and obtain collected data over the public network, multifactor authentication protocols for smart home have gained wide attention. However, most of these protocols cannot withstand impersonation attack, smart device lost attack, privileged-insider attack, smart card lost attack, and so on. Besides, high communication and computational costs weaken the system performance, which leads to most authentication protocols are not suitable for resource-constrained smart devices. To mitigate the aforementioned drawbacks, we proposed a PUF-assisted lightweight group authentication and key agreement protocol to implement secure access to multiple devices in the smart home simultaneously using the Chinese Remainder Theorem and secret sharing technique. Our protocol also utilizes physical unclonable function (PUF) and fuzzy extractor technique to extract the digital fingerprint of the smart devices, which can uniquely validate smart devices and protect the secrets stored in their memory. Our protocol can support various security features and withstand the many well-known attacks in the smart home. The performance analysis indicates that the proposed protocol can efficiently reduce communication/computational costs when the user simultaneously accesses multiple devices.