In the forming process of long threads by axial self-infeed rolling process, the protrusion generates on the top of the tooth profile, which affects the strength and precision of the formed threads. In this article, the principle of the axial self-infeed rolling process was described, and the finite element model was established to simulate the forming process of threads. The formation mechanism of protrusion was analyzed through material flow, which was verified by experiments. The pre-rolling angle and material’s influence on the protrusion size was also investigated. The results showed that the axial motion of the rollers squeezed one flank of the tooth, leading to higher protrusion on the face side as compared to the other side. During the rolling process, the height of the protrusion increased first and then decreased, reaching the maximum value when the height of the formed tooth was 73% of the whole tooth. In order to reduce the protrusion height, the pre-rolling angle should be 2°.