Objectives: To develop an efficient Video Summarization technique that aims to utilize the saliency map for mimicking the human way of selecting the important events in the given video. Methods: This paper proposes Histogram based Weighted Fusion (HWF) algorithm that uses spatial and temporal saliency maps to act as guidance in creating the summary of the video. The spatial saliency score and temporal saliency score obtained from the corresponding saliency maps are fused using the proposed HWF algorithm to obtain the frame level importance score. It tries to depict the visual attention of the human brain when watching a particular video. Findings: The experimental results show that the proposed HWF algorithm performs better than the state-of-the-art methods. Novelty: The use of Histogram intersection and the incorporation of the exponential function as the weight for the combined feature enhance the summarization ability of the proposed model. Keywords: Video Summarization, Saliency Map, Histogram intersection, Contrast sensitivity function, Attention curves