A certain number of hole-like defects will occur in aluminum alloys under cyclic loading. The internal holes will reduce the strength of the material and cause stress concentration, which will aggravate the development of fatigue damage. A classification method of defect features based on X-ray CT damage data is proposed. The random hole distribution model is established through the linear congruence method and the region division method. The hole parameter is introduced as the intermediate variable of the 3D reconstruction model of internal defects. In the mesoscopic stage, the function relationship between the distribution of random holes and the fatigue life is established based on the coupling relationship between the number and proportion of pores and the fatigue life. In the macroscopic stage, the relationship between the random holes and the macroscopic crack growth life is established by taking the crack length as the damage variable. The crack propagation rate decreased with the increase in the number of holes. The prediction model of the whole life stage is established using the life function from microcrack initiation to macroscopic crack propagation. Finally, the validity of the whole stage fatigue life prediction model is demonstrated through the comparison and verification of experiments, which provides a certain engineering value for the life estimation of 6061-T6 aluminum alloy materials.