The sustainable management of public hospitals is usually threatened by long-term operating deficit, which was exacerbated during the COVID-19 pandemic. This study aimed to quantitatively decompose the historical changes in the annual operating costs of public hospitals in Japan to identify the main driving forces responsible for a worsening imbalance between operating costs and income over the past two decades. A dataset of the annual operating costs of public hospitals in Japan was compiled, in which influencing factors were redefined to make the data amenable to the application of a decomposition method referred to as the Logarithmic Mean Divisia Index (LMDI). Using the LMDI method, the contribution of each influencing factor to the changes in public hospital operating costs was quantitatively determined. The results indicate that, on average, there is an annual reduction in operating costs by JPY 9 million per hospital, arising out of the national reform of public hospitals, but the rapid increase in the prices and worsened structure of costs in recent years resulted in an annual increment of JPY 127 million per hospital to the increasing operating costs. The pandemic revealed damage to the financial balance of public hospitals, but epidemic prevention policies brought an offset to the increased operating cost. A more resilient domestic medical supply chain, the introduction of new technologies, and continuous endeavors in system reform and pricing policies are required to achieve financial sustainability in public hospitals in Japan.