Physical inactivity is a major concern and poor adherence to exercise programs is often reported. The aim of this paper was to systematically review published reviews on the study of adherence to physical exercise in chronic patients and older adults and to identify those adherence-related key factors more frequently suggested by reviews for that population. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. Results were classified considering the target population and participants’ characteristics to identify the most repeated factors obtained for each condition. Fifty-five articles were finally included. Fourteen key factors were identified as relevant to increase adherence to physical exercise by at least ten reviews: (a) characteristics of the exercise program, (b) involvement of professionals from different disciplines, (c) supervision, (d) technology, (e) initial exploration of participant’s characteristics, barriers, and facilitators, (f) participants education, adequate expectations and knowledge about risks and benefits, (g) enjoyment and absence of unpleasant experiences, (h) integration in daily living, (i) social support and relatedness, j) communication and feedback, (k) available progress information and monitoring, (l) self-efficacy and competence, (m) participant’s active role and n) goal setting. Therefore, adherence to physical exercise is affected by several variables that can be controlled and modified by researchers and professionals.