Abstract. In this paper, we propose a non-parametric approach for shot boundary detection in videos. The proposed method exploits the split and merge framework by the use of color histograms. Initially, every frame of the input video sequence undergoes color quantization and subsequently, the color histograms are computed for every quantized frame. The split and merge is driven by the fishers linear discriminant criterion function which results with a set of subsequences after several iterations which are assumed to be the shots present in the given video. The proposed method is experimentally tested on video samples from TrecVid 2002 dataset and YouTube online database. We have obtained overall accuracy of 85.5% Precision, 87.1% Recall and 86.1% Fmeasure for the dataset used. A comparative study of the proposed approach with the contemporary research works is also carried out.