This paper considers video surveillance applied to traffic video streams. We present a framework for analyzing and recognizing different traffic behaviors from image sequences acquired from a fixed camera. Two types of interactions have been mainly considered. In one there is interaction between two or more mobile objects in the Field of View (FOV) of the camera. The other is interaction between mobile objects and static objects in the environment. The framework is based on two types of a priori information: (1) the contextual information of the camera's FOV, in terms of the description of the different static objects of the scene and (2) sets of predefined behavior scenarios, which need to be analyzed in different contexts. At present the system is designed to recognize behavior from stored videos and retrieve the frames in which the specific behaviors took place. We demonstrate successful behavior recognition results for pedestrian-vehicle interaction and vehicle-checkpost interactions.