Phase unwrapping is one of the major challenges in multiple branches of science that extract three-dimensional information of objects from wrapped signals. In several applications, it is important to extract the unwrapped information with minimal signal resolution degradation. However, most of the denoising techniques for unwrapping are designed to operate on the entire phase map to remove a limited number of phase residues, and therefore they significantly degrade critical information contained in the image. In this paper, we present a novel, smart, and automatic filtering technique for locally minimizing the number of phase residues in noisy wrapped holograms, based on the phasor average filtering (PAF) of patches around each residue point. Both patch sizes and PAF filters are increased in an iterative algorithm to minimize the number of residues and locally restrict the artifacts caused by filtering to the pixels around the residue pixels. Then, the improved wrapped phase can be unwrapped using a simple phase unwrapping technique. The feasibility of our method is confirmed by filtering, unwrapping, and enhancing the quality of a noisy hologram of neurons; the intensity distribution of the spatial frequencies demonstrates a 40-fold improvement, with respect to previous techniques, in preserving the higher frequencies.