In this paper, a new vibrational modal analysis technique was developed for intraoperative cementless prosthesis fixation evaluation upon hammering. Methods: An artificial bone (Sawbones)-prosthesis system was excited by sweeping of a sine signal over a wide frequency range. The exponential sine sweep technique was implemented to the response signal in order to determine the linear impulse response. Recursive Fourier transform enhancement (RFTE) technique was applied to the linear impulse response signal in order to enhance the frequency spectrum with sharp and distinguishable peak values indicating distinct high natural frequencies of the system (ranging from 15 kHz to 90 kHz). The experiment was repeated with 5 Sawbones-prosthesis samples. Upon successive hammering during the prosthesis insertion, variation of each natural frequency was traced. Results: Compared to classical Fast Fourier Transform, RFTE provided a better tracing and enhancement of frequency components during insertion. Three different types of frequency evolving trends (monotonically increasing, insensitive, and plateau-like) were observed for all samples, as confirmed by a new finite element simulation of the prosthesis dynamic insertion. Two main mechanical phenomena (i.e., geometrical compaction and compressive stress) were shown to govern these trends in opposite ways. Followup of the plateau-like trend upon hammering showed that the frequency shift is a good indicator of fixation. Conclusion: Alongside the individual follow-up of frequency shifts, combinatorial frequency analysis provides new objective information on the mechanical stability of Sawboneprosthesis fixation. Significance: The proposed vibrational technique based on RTFE can provide the surgeon with a new assistive diagnostic technique during the surgery by indicating when the bone-prosthesis fixation is acceptable, and beyond of which further hammering should be done cautiously to avoid bone fracture.