Milling carbon fiber reinforced polymer (CFRP) composites faces significant challenges, such as rapid tool wear leading to machining damage, even for a polycrystalline diamond (PCD) cutter. However, the effects of the milling conditions on the tool wear of PCD cutters remain unclear. This study investigated the effects of the cutting speed and fiber orientation on the tool wear of a PCD cutter and the corresponding machining quality in milling CFRP. These effects were analyzed by considering fiber and cutter interactions, and milling experiments with a PCD cutter were conducted for validation based on the analyses of cutting-edge profiles, cutting forces, burr area, and machined surface of CFRP. The worn cutting edge profiles were found to be elliptical in shape for cutting CFRP in different fiber orientations owing to the more severe wear on the flank face and cutting edge. Tool wear decreased with increasing cutting speed under a consistent feed per tooth, which further led to a reduction in the cutting force, burr area, and surface roughness. In addition, the maximum and minimum variations of worn cutting edge profiles were found in the cutting of 45° and 135° CFRP, respectively, suggesting that frequently changing the tool working position at 45° could suppress burr damage and increase tool life.