The motion of a relativistic test electron in a free electron laser can be altered significantly by the equilibrium self-field effects produced by the beam space charge and current and by the transverse spatial inhomogeneities in a realizable magnetic wiggler field. In a field configuration consisting of an ideal (constant-amplitude) helical wiggler field and a uniform axial guide field, it is shown that the inclusion of self-field effects destroys the integrability of the particle equations of motion. Consequently, the Group-I orbits and the Group-II orbits become chaotic at sufficiently high beam density. An analytical estimate of the threshold value of the self-field parameter F, = wg/ckw for the onset of chaoticity is obtained and found to be in good agreement with computer simulations. In addition, the effects of transverse spatial gradients in a realizable helical wiggler field with three-dimensional spatial variations are investigated in the absence of an axial guide field, but including self-field effects. For a thin electron beam (k 2r2 < 1) and small wiggler field amplitude (a, < yb 2 ), it is shown that the motion is regular and confined radially provided E, < ybaw/(1 + al). However, because of the intrinsic nonintegrability of the motion, the regular region in phase space diminishes in size as the wiggler amplitude is increased. The threshold value of the wiggler amplitude for the onset of chaoticity is estimated analytically and confirmed by computer simulations for the special case where self-field effects are negligibly small. Moreover, it is shown that the particle motion becomes chaotic on a time scale comparable with the beam transit time through one wiggler period.