The aim of this paper work is to design a user independent framework for recognizing and classifying the leaves in a video frames. This project involves classification of leaves using KNN (K- Nearest Neighbor) as a classifier. SURF (Speeded-Up Robust Features) and LBP (Local Binary Pattern) features are used for extracting Scale ,Orientation etc., In the first step our proposed model can extract most distinguish key-frames and then from extracted key-frames it detects the leaf color and recognize the different class of leaves.