The association between frailty, disability in activities of daily living (ADL), polypharmacy, and quality of life (QoL) in middle-aged patients with cardiovascular disease (CVD) is little investigated. This study sought (a) to explore this association comparatively in elderly and middle-aged hospitalized patients with CVD and (b) to determine which domains of ADL and QoL might improve the frailty prediction. A one-year follow-up study including 90 elderly (≥65 years old) and 89 middle-aged patients (40–65 years old) was conducted. At baseline, frailty assessment was performed based on the Fried criteria; Barthel Index (BI) and Duke Activity Status Index (DASI) were used for ADL, and European Quality of Life-5 dimensions (EQ-5D) for QoL. At follow-up, data were collected via telephone. At baseline, 79 patients (51 elderly and 28 middle-aged) were frail. The CVD frail patients showed functional dependency and a poor QoL compared to the non-frail (p < 0.001) and within each subgroup at follow-up. Mobility was found to predict frailty in both elderly (OR = 2.34) (C.I. (1.03–5.29)) and middle-aged patients (OR = 2.58) (C.I. (1.15–5.78)). The ADL assessment and self-reported QoL may help to identify an aggravation or an advanced frailty condition in hospitalized elderly and middle-aged CVD patients.