In this paper, we propose a new framework for detecting the unauthorized dumping of garbage in real‐world surveillance camera. Although several action/behavior recognition methods have been investigated, these studies are hardly applicable to real‐world scenarios because they are mainly focused on well‐refined datasets. Because the dumping actions in the real‐world take a variety of forms, building a new method to disclose the actions instead of exploiting previous approaches is a better strategy. We detected the dumping action by the change in relation between a person and the object being held by them. To find the person‐held object of indefinite form, we used a background subtraction algorithm and human joint estimation. The person‐held object was then tracked and the relation model between the joints and objects was built. Finally, the dumping action was detected through the voting‐based decision module. In the experiments, we show the effectiveness of the proposed method by testing on real‐world videos containing various dumping actions. In addition, the proposed framework is implemented in a real‐time monitoring system through a fast online algorithm.