Abstract. We propose a new regularization framework for inverse lithography that regularizes masks directly by applying a mask filtering technique to improve computational efficiency and to enhance mask manufacturability. This technique is different from the conventional regularization method that regularizes a mask by incorporating various penalty functions to the cost function. We design a specific mask filter for this purpose. Moreover, we introduce a metric called edge distance error (EDE) to guide mask synthesis and establish the correlation between pattern error and edge placement error (EPE) via EDE. We prove that EDE has the same dimension as EPE and has a continuous expression as pattern error. Simulation results demonstrating the validity and efficiency of the proposed method are presented. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.