The filament bundle is a kind of fiber assembly with an ultrahigh length–diameter ratio. For textiles, the filament is an intermediate product in the textile production process. For ease of storage, transportation and subsequent processing, the filament is wound to form a disc-shaped package with a certain size. During winding, the high-speed moving filament is placed precisely on the cylindrical surface of the package with the spiral shape by the traverse mechanism. Unfortunately, uneven tension of the filament during reversing is caused by the action of the traverse mechanism, which directly affects the quality of the filament package. In this article, the approach of modeling and simulating the filament winding system was proposed, of which the element model of the filament was established by the absolute node coordinate formulation, considering its viscoelasticity, as well as the effect of gravity. The contact model between the filament and the rotary vane, as well as the contact roller, was established by the Hertz law. The equations of motion for the system were derived by the Lagrange equations, and the algorithm for solving the differential-algebraic equations was further derived. Then, the dynamic behavior of the winding system during reversing was simulated and analyzed, and the results show that the moving filament is in non-contact with the rotary vane at the moment of reversing, and the small fluctuation in contact force between the rotary vane and the filament causes the large fluctuation in the filament tension. Finally, the validity is tested through an experiment.