The phase stability of an optical coherence elastography (OCE) system is the key determining factor for achieving a precise elasticity measurement, and it can be affected by the signal-to-noise ratio (SNR), timing jitters in the signal acquisition process, and fluctuations in the optical path difference (OPD) between the sample and reference arms. In this study, we developed an OCE system based on swept-source optical coherence tomography (SS-OCT) with a common-path configuration (SS-OCE CP). Our system has a phase stability of 4.2 mrad without external stabilization or extensive post-processing, such as averaging. This phase stability allows us to detect a displacement as small as 300 pm. A common-path interferometer was incorporated by integrating a 3-mm wedged window into the SS-OCT system to provide intrinsic compensation for polarization and dispersion mismatch, as well as to minimize phase fluctuations caused by the OPD variation. The wedged window generates two reference signals that produce two OCT images, allowing for averaging to improve the SNR. Furthermore, the electrical components are optimized to minimize the timing jitters and prevent edge collisions by adjusting the delays between the trigger, k-clock, and signal, utilizing a high-speed waveform digitizer, and incorporating a high-bandwidth balanced photodetector. We validated the SS-OCE CP performance in a tissue-mimicking phantom and an in vivo rabbit model, and the results demonstrated a significantly improved phase stability compared to that of the conventional SS-OCE. To the best of our knowledge, we demonstrated the first SS-OCE CP system, which possesses high-phase stability and can be utilized to significantly improve the sensitivity of elastography.