Motivated by the quest for decreased healthcare costs and further fueled by the COVID pandemic, wearable devices have gained major attention in recent years. Yet, their secure usage and patients' privacy continue to be concerning. To address these issues, the paper presents SWeeT, a novel lightweight protocol for allowing flexible and secure access to the collected data by multiple caregivers while sustaining the patient's privacy. Particularly, SWeeT deploys Physically Unclonabale Functions (PUFs) to generate encryption keys to safeguard the patients' data during transmission. The computation overhead is significantly reduced by applying very simple encryption operations while enabling frequent change of the keys to sustain robustness. SWeeT is shown to counter impersonation, Sybil, man-in-the-middle, and forgery attacks. SweeT is validated through experiments using implementation on an Artix7 FPGA and through formal security analysis. 6