In our paper, we have proposed to use graphs to detect anomaly in human action video. Although the detection of anomaly is a widely researched topic, but very few researchers have detected anomaly in action video using graphs. in our proposed method we have represented the smaller section (sub-section) of our input video as a graph where vertices of the graph are the space time interest points in the sub-section video and the association between the space time interest points exists. Thus, graphs for each sub section are created to look for a repeated substructure. We believe most of the actions inherently are repeated in nature. Thus, we have tried to capture the repetitive sub-structure of the action represented as a graph and used this repetitive sub-structure to compress the graph. If the compressed graph has few elements that have not been compressed, we suspect them as anomaly. But the threshold value takes care not to make the proposed method very much sensitive towards the few uncompressed elements. Our proposed method has been implemented on locally created “extended KTH” and “extended Weizmann” datasets with good accuracy score. The proposed method can also be extended for few more applications such as training athletes and taking elderly care.