This paper proposes an integrated unified elasto-viscoplastic fatigue and creep damage model with modified hardening equations to simulate the elasto-viscoplasticity, primary-secondary and tertiary creep, relaxation, cyclic softening, temperature-dependency, and creep-fatigue interaction (CFI) damage. For this purpose, we integrate the creep damage and fatigue damage model into the viscoplasticity-damage model to predict tertiary creep and cyclic softening behavior caused by creep and fatigue damage modes. Interaction effects between each damage mode are considered by the creep-fatigue interaction (CFI) damage scheme. This paper also introduces a genetic algorithm-based integrated characterization method that can effectively optimize material input parameters of the unified elasto-viscoplastic-damage material model with limited experimental data. Implicit Euler's backward iteration scheme is adopted to improve the convergence and accuracy of solutions. The proposed material model is implemented in UMAT for finite element (FE) analysis of high-temperature structures under cyclic-dwell loading. Experimental data of tensile, dwell, creep, and cyclic loading at various temperatures are used to verify the model. Finally, full-scale turbine blade FE analysis is performed under anisothermal conditions. The proposed material modeling framework can also be utilized for other high-temperature structures, such as power plants, leading-edge nose cones, etc.