Video surveillance is one of the major applications where high-resolution (HR) images are crucial. Since the video camera has limited spatial and temporal resolution, there is a need for super resolution video generation algorithms. In this paper, we have presented a novel technique for activity detection in the surveillance video. To achieve this goal, we have proposed and investigated efficient algorithms for Video Object Plane (VOP) generation, shadow removal from VOP and super-resolved VOP generation, for activity detection from surveillance video. The proposed VOP generation algorithm is computationally efficient and works for both dynamic and static backgrounds. The novel shadow removal algorithm for the VOP is based on texture and its performance has been studied based on average shadow detection and discrimination rates. The proposed super-resolution video generation algorithm has been designed using edge models. The performance of this algorithm has been evaluated using a numerical analysis technique and is found to be better than bi-cubic and bi-linear interpolation techniques.