Silicon carbide fiber reinforced silicon carbide ceramic matrix composite (SiCf/SiC composite) is characterized by a high strength-to-density ratio, high hardness, and high temperature resistance. However, due to the brittleness of the matrix material and the anisotropy of the reinforcing phase, it is a huge challenge for machining of the material. The milling method has advantages of a high material removal rate and applicability to complex surface geometry. However, no published literature on milling of SiCf/SiC composite has been found up to now. In this paper, high-speed milling of SiCf/SiC composites was carried out under dry conditions and cryogenic cooling using liquid nitrogen, respectively. Polycrystalline diamond (PCD) and chemical vapor deposition (CVD) diamond cutting tools were used for the milling work. The cutting performance of the two kinds of tools in high-speed milling of SiCf/SiC composites was studied. Tool failure modes and mechanisms were analyzed. The effects of the cooling approach on tool wear and machined surface quality were also investigated. The experimental results showed that under identical cutting parameters and cooling approaches, the PCD tool yielded better cutting performance in terms of a longer tool life and better surface quality than that of the CVD diamond tool. In dry machining, the failure modes of the CVD diamond tool were a large area of spalling on the rake face, edge chipping and severe tool nose fracture, whereas for the PCD tool, only a small area of spalling around the tool nose took place. Compared to the dry machining, the wear magnitudes of both PCD and CVD diamond tools were decreased in cryogenic machining. Additionally, the surface quality also showed significant improvements. This study indicates that the PCD tool is highly suitable for machining of SiCf/SiC composite, and that the cryogenic method can improve machining efficiency and surface quality.