The semiconductor manufacturing process is a complex and highly technical operation, demanding precise and consistent metrology throughout its prolonged duration. As manufacturing processes grow in complexity, it becomes imperative to measure key variables of the intermediate products. Consequently, there is an increased demand for higher throughput in metrology to enhance measurement capacity. The electron-beam (E-Beam) metrology tools can be accelerated by integrating denoiser models for image enhancement and reducing the sample rate of raw images. We propose a novel universal denoising method specifically tailored for the semiconductor industry. Distinct from traditional denoisers designed for standard cameras and the RGB space, our proposed solution is tailored to the nano-scale structures and noise patterns inherent in semiconductor images obtained via E-beam tools. To meet the strict precision requirements of semiconductor metrology, where even minimal deviations carry substantial implications, our approach introduces a specialized network structure with novel loss functions that reflect the unique characteristics of semiconductor metrology. We propose a novel conditioning scheme to apply a single trained model to diverse image domains from various wafer products and layers. By combining the proposed loss functions and conditioning scheme, we can universally handle multiple image domains with a single model. This method significantly reduces time expenditure while preserving the crucial accuracy necessary for high-quality semiconductor production. Through comprehensive volume testing across diverse metrology recipes, which covers the entire spectrum of DRAM fabrication, our method has demonstrated a notable increase in metrology throughput, achieving an average enhancement of 26%, and reaching up to 46%, all without compromising accuracy or reliability. This breakthrough offers a versatile and efficient solution, marking a significant advancement in the field of semiconductor manufacturing.