Human tracking is one of the challenging and essential components of an intelligent surveillance system. Variety of correlation filter-based tracking algorithms has presented their brilliance in human tracking. However, few of them have demonstrated disappointments in the presence of occlusion, background clutters, illumination variation, scale variation, fast motion, in-plane rotation, and out of the plane rotation. The paper presents, an improved correlation filter-based tracking algorithm, harmonious polling of patched correlation technique for tracking a human in a video sequence thinking about all the challenging attributes. A human to be tracked is represented by using multiple image patches, as patch-based tracking framework resolves the issues based on occlusion and global scene changes to a great extent. An innovative methodology utilized in the proposed framework is, every individual patch is treated independently thought the process and applied to the polling mechanism, which improves the performance of the system to an extraordinary degree. Kernelized correlation filter is applied to each patch individually generating the correlation score. A polling mechanism is a novel technique used in the proposed framework, which generates the confidence map from the correlation score. The maximum score achieved from the confidence map gives an exact position of the target. The tracker is applied to the number of challenging sequences and contrasted with various outperforming algorithms. The precision and success rate of the proposed tracker is improved by 15% and 19%, respectively. From the qualitative and quantitative analysis, it can be demonstrated that the proposed algorithm beats the cutting-edge execution.