Detecting and discriminating humans in video frames for surveillance applications is a demanding task. Identifying and highlighting humans by eliminating shadows from the video frames is vital for prudence motive. In this paper, a three-step procedure is proposed, which includes motion detection by background subtraction in live video frames, morphological gradient-based shadow removal, and human detection by Hybrid Feature Set (HFS), which comprises Histogram Oriented Gradient (HOG) and Local Binary Pattern (LBP) with adaptive Neuro-Fuzzy inference system. The first step incorporates static background subtraction and dynamic background subtraction. The second step is to remove shadows by using a morphological gradient with the horizontal directional mask. The third step includes near-field, mid-field, and far-field human detection by using an adaptive Neuro-Fuzzy inference system. The results obtained from the various performed experimental analysis demonstrates diverse parametrical measures, which outperforms comparatively when benchmark databases and real-time surveillance video frames were used.