A mass of video data is processed at the control center in most video surveillance systems, this brings some difficulty in data integration and real-time computation. In this paper we present a hierarchical structure of two-tier processing. The camera is designed as a DSP-based terminal system, there will run algorithms to analysis video data to get the feature of moving object. Then the feature data is processed in monitoring center further. The hardware core of embedded terminal is TMS320DM642 chip, we describe the video processing algorithms and the software modules in embedded terminal. This hierarchical design is provided for improving real-time performance and reliability of video data processing., Further more, it could be easier to intellectualize video surveillance system.