Background Ethiopia's elderly care service is relatively young and little is known about the practices or readiness of nurses to provide care. Nurses need to have good knowledge, a positive attitude, and experience when it comes to caring for the elderly or patients who are chronically providing quality care. This study aimed to assess the knowledge, attitudes, and practices towards care for elderly patients and associated factors among nurses working in adult care units of public hospitals in Harar, 2021. Methods An institutional-based descriptive cross-sectional study was conducted from February 12 to July 10, 2021. A simple random sampling technique was used to select 478 study participants. Data were collected by trained data collectors using a pretested self-administered questionnaire. From the pretest, Cronbach's alpha for all items was above 0.7 for all items. Data were entered into Epi Data v.4.6 and exported to the Statistical Package for Social Science Version 26 for binary logistic regression analysis. A P-value of 0.05 was used to declare a significant association between variables. Results The study revealed that 311 (69%) had inadequate knowledge. Having a first degree and an unfavorable attitude towards nurses showed a statistically significant association with nurses’ inadequate knowledge. A total of 275 (61.0%) nurses were found to have an unfavorable attitude and were significantly associated with having a diploma and first degree, learning in a private organization, 6 to 10 years of experience, lack of training, and inadequate knowledge of nurses. A total of 297 (65.9%) study units had inadequate practice towards the care of elderly patients. Nurses’ practices showed a significant association with the type of hospital, work experience, and adherence to guidelines with a response rate of 94.4%. Conclusion The majority of nurses had inadequate knowledge, an unfavorable attitude, and inadequate practice towards the care of elderly patients. Having a first-degree and an unfavorable attitude with inadequate knowledge, lack of training and inadequate knowledge, unfavorable attitudes and less than 11 years of experience, working in non-academic hospitals, and unavailability of guidelines with inadequate practices were significantly associated.