Complex eigenvalue analysis has generally been applied to squeal improvement of automotive brake systems in recent years. Discrimination of the occurrence of unstable vibration by modal coupling has become common in brake design. However, the generation mechanism is not fully understood. In particular, the transition of the eigenvalue and the bifurcation phenomenon accompanying the change of the equation of motion are difficult to predict quantitatively. One reason is that a degree of instability can be expressed by real parts of eigenvalues, but the difficulty of the coupling cannot be determined by real parts of eigenvalues. In this report, to obtain a stability index before the coupling, characteristic polynomials are the subject of this research. For the structural instability problem of a two-degree-of-freedom system, which involves a typical friction-induced vibration, the positions of the eigenvalues are geometrically shown on a complex response surface using a real part of the characteristic polynomial. The characteristic polynomial of the two-degree-of-freedom system becomes a complex quartic function; as a result, a saddle-shaped response surface that intersects the complex space appears. Without damping, when the surface and a zero plane of the complex space intersect on an imaginary axis, uncoupled eigenvalues appear. When the surface does not intersect on the imaginary axis, the eigenvalues become complex, and unstable vibration occurs. In a parameter study, a friction coefficient raises the surface monotonously, while mass and stiffness change the eigenvalues in the same way as curve veering, and damping breaks the symmetry of the surface and shifts the surface to the stable side. The vertex in the imaginary axis cross section of the surface is a point at which the eigenvalues reach the coupling. This point serves as a guideline for the stability before the coupling. Consequently, it is useful for evaluating the transition of the eigenvalues.