In this paper, we address the linear stability analysis of a confined potential vortex with a free surface. This particular flow h as b een r ecently u sed b y Tophøj et al. (Phys. Rev. Lett., vol. 110(19), 2013, article 194502) as a model for the swirling flow o f fl uid in an op en cy lindrical co ntainer, dr iven by ro tating the bottom plate (the rotating bottom experiment) to explain the so-called rotating polygons instability (Vatistas J. Fluid Mech., vol. 217, 1990, pp. 241-248; Jansson et al., Phys. Rev. Lett., vol. 96, 2006, article 174502) in terms of surface wave interactions leading to resonance. Global linear stability results are complemented by a Wentzel-Kramers-Brillouin-Jeffreys (WKBJ) analysis in the shallow-water limit as well as new experimental observations. It is found that global stability results predict additional resonances that cannot be captured by the simple wave coupling model presented in . Both the main resonances (thought to be at the root of the rotating polygons) and these secondary resonances are interpreted in terms of over-reflection p henomena b y t he W KBJ a nalysis. F inally, w e p rovide experimental evidence for a secondary resonance supporting the numerical and theoretical analysis presented. These different methods and observations allow to support the unstable wave coupling mechanism as the physical process at the origin of the polygonal patterns observed in free-surface rotating flows.