This paper investigates a problem of passive mitigation of vibratory instabilities caused by two unstable modes by means of a single nonlinear energy sink (NES). For this purpose, a linear four-degree-of-freedom (DOF) primary structure having two unstable modes (reproducing the typical dynamic behavior of a friction system) and undergoing, as it is linear, unbounded motions when it is unstable, is coupled to a NES. In this work, the NES involves an essentially cubic restoring force and a linear damping force. We are interested in characterizing analytically the response regimes resulting from the coupling of the two unstable linear modes of the primary structure and the nonlinear mode of the NES. To this end, from a suitable rescaling of the governing equations of the coupled system in which the dynamics of the primary structure is reduced to its unstable modal coordinates, the complexification-averaging method is applied. The resulting averaged system appears to be a fast-slow system with four fast variables and two slow ones related to the two unstable modes of the primary structure. The critical manifold of the averaged dynamics is obtained through the geometric singular perturbation theory and appears as a two-dimensional parametric surface (with respect to two of the four fast variables) which evolves in the whole six-dimensional variable space. The asymptotic analysis reveals that the NES attachment can produce some bounded responses and suggests that the system may have simultaneous stable attractors. Numerical simulations complement the study, highlighting a possible competition between stable attractors and allowing us to investigate their basins of attraction. In each considered situation, a good agreement has been observed between theoretical results and numerical simulations, which validates the proposed asymptotic analysis.