Ball-end cutters are widely used for machining the parts of Ti-6Al-4V, which have the problem of poor machined surface quality due to the low cutting speed near the tool tip. In this paper, through the experiments of inclined surface machining in different feed directions, it is found that the surface adhered damages will form on the machined surface under certain tool postures. It is determined that the formation of surface adhered damage is related to the material adhesion near the cutting edge and the cutting-into/out position within the tool per-rotation cycle. In order to analyze the cutting-into/out process more clearly under different tool postures, the projection models of the cutting edge and the cutter workpiece engagement on the contact plane are established; thus, the complex geometry problem of space is transformed into that of plane. Combined with the case of cutting-into/out, chip morphology, and surface morphology, the formation mechanism of surface adhered damage is analyzed. The analysis results show that the adhered damage can increase the height parameters Sku, Sz, Sp, and Sv of surface topographies. Sz, Sp, and Sv of the normal machined surface without damage (Sku ≈ 3) are about 4–6, 2–3, and 2–3 μm, while Sz, Sp, and Sv with adhered damage (Sku > 3) can reach about 8–20, 4–14, and 3–6 μm in down-milling and 10–25, 7–18, and 3–7 μm in up-milling. The feed direction should be selected along the upper left (Q2: β∈[0°, 90°]) or lower left (Q3: β∈[90°, 180°]) to avoid surface adhered damage in the down-milling process. For up-milling, the feed direction should be selected along the upper right (Q1: β∈(−90°, 0°]) or upper left (Q2: β∈[0°, 90°)).