The global rise in the elderly population, which presents challenges to healthcare systems owing to labor shortages in caregiving facilities, necessitates innovative solutions for elderly care services. Smart aging technologies such as robotic companions and digital home gadgets, offer a solution to these challenges by improving the elderly's quality of life and assisting caregivers. However, limitations in data privacy, real-time processing, and reliability often hinder the effectiveness of the existing technologies. Among these, privacy concerns are a major barrier to ensuring user trust and ethical implementation. Therefore, this study proposes a more effective approach for smart aging through elderly activity monitoring that prioritizes data privacy. The proposed system utilizes stereo depth cameras to monitor the activities of the elderly. Data were collected from real-world environments with the participation of six elderly individuals from a care center and hospital. This system focuses on recognizing common daily actions of the elderly including sitting, standing, lying down, and seated in a wheelchair. Additionally, it recognizes transition states (in-between actions such as changing from sitting to standing) that are crucial for assessing balance issues. By integrating motion information with a deep-learning architecture, the system achieved a high accuracy of 99.42% in recognizing daily actions in real-time. This high accuracy was maintained even with minimal data from new environments through transfer learning, and the adaptability of this model ensured its potential for real-world applications. For intuitive interaction between the caregivers and the system, a user-friendly graphical interface (GUI) was also designed in the proposed approach.