Stochastic thermodynamics provides an important framework to explore small physical systems where thermal fluctuations are inevitable. In the studies of stochastic thermodynamics, some thermodynamic quantities, such as the trajectory work, associated with the complete Langevin equation (the Kramers equation) are often assumed to converge to those associated with the overdamped Langevin equation (the Smoluchowski equation) in the overdamped limit under the overdamped approximation. Nevertheless, a rigorous mathematical proof of the convergence of the work distributions to our knowledge has not been reported so far. Here we study the convergence of the work distributions explicitly. In the overdamped limit, we rigorously prove the convergence of the extended Fokker-Planck equations including work using a multiple timescale expansion approach. By taking the linearly dragged harmonic oscillator as an exactly solvable example, we analytically calculate the work distribution associated with the Kramers equation, and verify its convergence to that associated with the Smoluchowski equation in the overdamped limit. We quantify the accuracy of the overdamped approximation as a function of the damping coefficient. In addition, we experimentally demonstrate that the data of the work distribution of a levitated silica nanosphere agrees with the overdamped approximation in the overdamped limit, but deviates from the overdamped approximation in the low-damping case. Our work fills a gap between the stochastic thermodynamics based on the complete Langevin equation (the Kramers equation) and the overdamped Langevin equation (the Smoluchowski equation), and deepens our understanding of the overdamped approximation in stochastic thermodynamics.