Human activity recognition systems (HARSs) are vital in a wide range of real-life applications and are a vibrant academic research area. Although they are adopted in many fields, such as the environment, agriculture, and healthcare and they are considered assistive technology, they seem to neglect the aspects of security and privacy. This problem occurs due to the pervasive nature of sensor-based HARSs. Sensors are devices with low power and computational capabilities, joining a machine learning application that lies in a dynamic and heterogeneous communication environment, and there is no generalized unified approach to evaluate their security/privacy, but rather only individual solutions. In this work, we studied HARSs in particular and tried to extend existing techniques for these systems considering the security/privacy of all participating components. Initially, in this work, we present the architecture of a real-life medical IoT application and the data flow across the participating entities. Then, we briefly review security and privacy issues and present possible vulnerabilities of each system layer. We introduce an architecture over the communication layer that offers mutual authentication, solving many security and privacy issues, particularly the man-in-the-middle attack (MitM). Relying on the proposed solutions, we manage to prevent unauthorized access to critical information by providing a trustworthy application.