The ability to detect and track object of interest from sequence of frames is a crit ical and vital problem of many vision systems developed as yet. This paper presents a smart surveillance system that tracks objects of interest in a sequence of frames in their own defined respective boundaries. The objects of interest are registered or saved within the system. We have proposed a unique tracking algorith m using co mbination of SURF feature matching, Kalman filtering and temp late matching approach. Moreover, an efficient technique is proposed that is used to refine reg istered object image, ext ract object of interest and remove extraneous image area fro m it. The system will track reg istered objects in their respective boundaries using real time v ideo generated through two IP cameras positioned in front of each other.