Flutter stability is a dominant design constraint of modern gas and steam turbines. To further increase the feasible design space, flutter-tolerant designs are currently explored, which may undergo Limit Cycle Oscillations (LCOs) of acceptable, yet not vanishing, level. Bounded self-excited oscillations are a priori a nonlinear phenomenon, and can thus only be explained by nonlinear interactions such as dry stick-slip friction in mechanical joints. The currently available simulation methods for blade flutter account for nonlinear interactions, at most, in only one domain, the structure or the fluid, and assume the behavior in the other domain as linear. In this work, we develop a fully-coupled nonlinear frequency domain method which is capable of resolving nonlinear flow and structural effects. We demonstrate the computational performance of this method for a state-of-the-art aeroelastic model of a shrouded turbine blade row. Besides simulating limit cycles, we predict, for the first time, the phenomenon of nonlinear instability, i.e. , a situation where the equilibrium point is locally stable, but for sufficiently strong perturbation (caused e.g. by an impact), the dry frictional dissipation cannot bound the flutter vibrations. This implies that linearized theory does not necessary lead to a conservative design of turbine blades. We show that this phenomenon is due to the nonlinear contact interactions at the tip shrouds, which cause a change of the vibrational deflection shape and frequency, which in turn leads to a loss of aeroelastic stability. Finally, we extend the well-known energy method to capture these effects, and conclude that it provides a good approximation and is useful for initializing the fully-coupled solver.