The low‐latitude boundary layer stability problem in the geotail is solved numerically. This study relies on a cylindrical model of the geotail, with a smooth boundary, enwrapped by a helical solar wind flow. It is shown that three types of unstable magnetohydrodynamic waves exist in this plasma system: (1) surface waves at the magnetopause, (2) waves radiated into the solar wind, and (3) eigenmodes of the waveguide within the geotail. Unstable surface waves generated in the low‐ to medium‐speed solar wind have the largest growth rate. They are driven in the Pc3–Pc6 geomagnetic pulsation range and have a local maximum at Pc4 frequencies. In the cylindrical model in question the minimum transverse wavelengths of the unstable oscillations are much less than the shear layer thickness. This differentiates it from models with Cartesian geometry, where the minimum transverse wavelengths of unstable oscillations are larger than the boundary layer thickness. In high‐speed solar wind flows the magnetopause is stable to surface waves but unstable to radiative modes. The growth rate of such oscillations is an order of magnitude smaller than that for surface waves generated in the low‐ to medium‐speed solar wind. If the helicity of high‐speed solar wind flows enwrapping the magnetosphere is small, the oscillations are unstable in the Pc4–Pc6 geomagnetic pulsation range, with their maximum growth rate being in the Pc5 range. If the helicity of high‐speed solar wind flow enwrapping the magnetosphere is large enough, the oscillations are unstable throughout the Pc1–Pc6 geomagnetic pulsation range.