Abstract. The behavior recognition is one of the most prolific lines of research in recent decades in the field of computer vision. Within this field, the majority of researches have focused on the recognition of the activities carried out by a single individual, however this paper deals with the problem of recognizing the behavior of a group of individuals, in which relations between the component elements are of great importance. For this purpose it is exposed a new representation that concentrates all necessary information concerning relations two to two present in the group, and the semantics of the different groups formed by individuals and training (or structure) of each one of them. The work is presented with the dataset created in CVBASE06 dealing the European handball