Damage to carbon nanotubes (CNTs) during the fabrication process of CNT reinforced composites has great influence on their mechanical properties. In this study, the 2014 Al with powder sizes of 20, 9 and 5 μm was selected to study the effect of initial particle size on the damage to carbon nanotubes (CNTs) during ball milling. The result shows that for CNTs in the ball milled CNT/Al (with powder size of 20 and 9 μm) mixtures, the intensity ratio of the D band and the G band (ID/IG) first increased and then reached a plateau, mainly because most of the CNTs are embedded, to a certain extent, in the aluminum powder after milling, which could protect the CNTs from damage during further milling. While for CNTs in the ball milled CNT/Al (with powder size of 5 μm) mixture, the ID/IG ratio continues to climb from 1.31 to 2.33 with time, indicating continuous damage to the CNTs occurs during the milling. Differential scanning calorimetry (DSC) analysis demonstrates that the chemical instability increased with an increase in the damage level of CNTs, resulting in the formation of aluminum carbide (Al4C3) at a lower temperature before the melting of aluminum, which is detrimental to their mechanical properties.