In phase-shifting digital holographic interferometry for measuring a displacement distribution of an object, holograms and reconstructed images have speckle noise and they provide large error in the calculation of displacement analysis. In order to decrease the effect of speckle noise, we previously proposed a new method using windowed holograms. In this paper, we propose a new averaging method of the obtained phase-difference values. Many phase-difference values at a point obtained by different windows for a hologram are averaged by considering the weight for each phase value. The weight is changed as the m-th power of the absolute amplitude of the complex amplitude of the reconstructed object. As the result, when the number n of the windowed holograms becomes larger, the standard deviation of the error becomes smaller. When the power m is 2, the error becomes the minimum. The standard deviation of the errors in the case of a flat plate with 316 nano-meter out-of-plane displacement is 88 pico-meter when n=1024 and m=2.