Recognizing human action in sports is difficult task as various sequences of activities involved in every scene. Identifying each action individually without overlapping of movements is a tedious process due to continuous change of frames within short duration. So proper tracking of human movements for each action is important. Hence new structure-based human action recognition and tracker technique (HART) is proposed. It uses joint trajectory images and visual feature to design each human action. At first, a structural based method employed to extract human skeleton data points from RGB (Red Green Blue) videos. Next, a Multitude Object Tracker (MOT) is proposed which uses the trajectory of human skeleton joints in an image space for identification of actions. Then, Histogram of Oriented Gradients (HOG) combined with Support Vector Machine (SVM) is applied to extract physical body shape and action information. Finally, the action label and interconnected keypoints in humans is jointly detected as end result. The proposed HART technique effectively performed well with the accuracy of about 82% over the other activity recognition methods.