Squeak is an unwanted, annoying noise generated by self-excited, friction-induced vibration. A unique squeak test apparatus that can generate squeak noises consistently was developed by modifying and employing a sprag-slip mechanism. Such an apparatus enables building database that accurately ranks squeak propensity of material pairs and will be highly useful for noise, vibration, and harshness (NVH) engineers and vehicle interior designers. An analytical model of the apparatus was developed to identify instability conditions that induce unstable, large-amplitude vibration, therefore squeak noises. A finite element model was established and studied in this work to refine the design of the apparatus and better understand underlying phenomena of the squeak generation. Complex eigenvalue analysis (CEA) was used to study the instability of the system and results show that the instability occurs by the coalescence of two modes, which makes the effective damping of one of the coalesced modes negative. The instability condition from the CEA shows good agreement with the results obtained from the analytical model. Furthermore, dynamic transient analysis (DTA) was performed to investigate the stability of the system and confirm the instability conditions identified from the CEA. The effects of main design parameters on the stability were investigated by DTA. The results obtained from the actual tests show that the test apparatus consistently generates unstable vibration of a very large amplitude, indicating generation of squeak noises.