In this paper we present a study of beam halo based on a three-dimensional particle-core model of an ellipsoidal bunched beam in a constant focusing channel including the effects of nonlinear rf focusing. For an initially mismatched beam, three linear envelope modes -a high frequency mode, a low frequency mode, and a quadrupole mode-are identified for an azimuthally symmetric bunched beam. The high frequency mode has three components all in phase; the low frequency mode has the transverse components in phase and the longitudinal component 180 ± out of phase; the quadrupole mode has no longitudinal component, and the two transverse components in the mode are 180 ± out of phase. We also study the case of an ellipsoidal bunched beam without azimuthal symmetry and find that the high frequency mode and the low frequency mode are still present but the quadrupole mode is replaced by a new mode with transverse components 180 ± out of phase and a nonzero longitudinal component. Previous studies, which generally addressed the situation where the longitudinal-to-transverse focusing strength is roughly 0.6 or less, conclude that the oscillation of the high frequency mode is predominantly transverse, and that of the low frequency mode is predominantly longitudinal. In this paper we present a systematic study of the features of the modes as a function of the longitudinal-to-transverse focusing strength ratio. We find that, when the ratio is greater than unity, the high frequency mode may contain a significant longitudinal component. Thus, excitation of the high frequency mode in this situation can be responsible for the formation of longitudinal beam halo. Furthermore, while previous studies have observed halo amplitudes roughly 2-3 times the matched beam edge, for the present parameters we observe much larger amplitudes (5 times or more). This is due to the fact that the longitudinalto-transverse focusing ratio used here is greater than that of previous studies. The finding of large transverse halo amplitude can have significant impact on the design of high-intensity ion accelerators where the longitudinal-to-transverse focusing ratio is slightly greater than unity in some parts of the linac.