Use is made of the Vlasov-Maxwell equations to investigate detailed properties of the sideband instability for a helical wiggler free-electron laser with wiggler wavelength A, = 27r/ko =const and normalized wiggler amplitude a, = eB./mc 2 ko =const. The model describes the nonlinear evolution of a right-circularly polarized primary electromagnetic wave with frequency w,, wavenumber k,, and slowly varying amplitude a,(z, t) and phase 8,(z,t) (eikonal approximation). The coupled Vlasov and field-evolution equations are analyzed in the ponderomotive frame ("primed" variables) moving with velocity v, = w,/(k, -+-ko) relative to the laboratory. Detailed properties of the sideband instability are investigated for small-amplitude perturbations about a quasi-steady state characterized by an equilibrium electron distribution f'(-yb) and a primary electromagnetic wave with constant amplitude a' =const (independent of z' and t') and slowly varying phase 60(z'). A formal dispersion relation is derived for perturbations about a general equilibrium distribution f 0 (ys) which may include both trapped and untrapped electrons. For the case where only trapped electrons are present, the dispersion relation is reduced to a simple analytical form. Detailed properties of the sideband instability are investigated for the case where the trapped electrons uniformly populate the ponderomotive potential up to an energy 7's < +, where j is the energy at the separatrix. Analysis of the dispersion relation shows that the maximum energy of the trapped-electron population (4') significantly affects detailed stability properties in the strong-pump and intermediate-pump regimes.