SUMMARYThe investigation of the extrusion swelling mechanism of viscoelastic fluids has both scientific and industrial interest. However, it has been traditionally difficult to afford theoretical and experimental researches to this problem. The numerical methodology based on the penalty finite element method with a decoupled algorithm is presented in the study to simulate three‐dimensional extrusion swelling of viscoelastic fluids flowing through out of a circular die. The rheological responses of viscoelastic fluids are described by using three kinds of differential constitutive models including the Phan‐Thien Tanner model, the Giesekus model, and the finite extensible nonlinear elastic dumbbell with a Peterlin closure approximation model. A streamface‐streamline method is introduced to adjust the swelling free surface. The calculation stability is improved by using the discrete elastic‐viscous split stress algorithm with the inconsistent streamline‐upwind scheme. The essential flow characteristics of viscoelastic fluids are predicted by using the proposed numerical method, and the mechanism of swelling phenomenon is further discussed.Copyright © 2013 John Wiley & Sons, Ltd.