Industrial boilers cause significant energy wastage that could be mitigated with oxy-fuel combustion versus traditional air combustion. Despite several feasibility studies on oxy-fuel burners, they are widely avoided in industry due to major infrastructural challenges. This study measured the performance and heat transfer characteristics of each component in a 0.5 MW fire tube gas boiler after retrofitting it with an oxy-fuel burner. Comparisons were drawn across three combustion modes—air combustion, oxy-fuel combustion, and oxy-fuel flue gas recirculation (FGR). The Dittus–Boelter equation was employed to predict heat transfer in the fire tube for all combustion modes at full load (100%). Heat transfer in the latent heat section of the economizer was measured and compared with predictions using the Zukauskas equation. With this retrofit, oxy-fuel combustion improved the thermal efficiency by about 3–4%. In oxy-fuel combustion, the flow rate of exhaust gas decreased. When integrated into an existing fire tube boiler, the fire tube’s heat transfer contribution diminished greatly, suggesting the economic viability of a redesigned, reduced fire tube section. Additionally, a new design could address the notable increase in gas radiation from the fire tube in oxy-fuel and FGR, as well as aid in the efficient recovery of condensation heat from exhaust gases.