Video surveillance renders great services to humans formonitoring the children, crime prevention and monitoring the patients. Yet, detecting the presence of human objects in the surveillance video is found to be hectic challenge due to various constraints, such as the view point of the camera, illumination and occlusion. In this research, a hybrid model is proposed to restrain the issues, like illumination and occlusions for effective recognition of human objects in the surveillance video. In this proposed hybrid model, the object in the frame are tracked with the aid of the rectangular bounding box, proposed search source-based Deep LSTM, and modified deep sort algorithm, where the effect of vanishing gradient is minimized and performance is boosted. The search space algorithm trains the Deep LSTM and establishes an effective trade-off between the exploration and exploitation phases with a better hyperparameters that impacts directly on the tracking performance. At the same time, the modified deep sort algorithm solves the assignment problem through incorporating the motion information, which in turn increases the tracking efficiency. The performance evaluation and comparative analysis are executed to demonstrate the effectiveness of the proposed Hybrid model. The proposed Hybrid model shows impressive results in terms of MOTA, MOTP and the tracking error of98.5%, 98.1% and 0.0135 respectively.