In this paper, we describe a cost-efficient programmable video signal processor (PVSP) to implement various video encoding and decoding schemes. Hierarchical (two levels) programmable control architecture, flexible memory address mapping strategies and a p r o g m a b l e VLCNLD module are applied in order to achieve sufficient programmability. Thus, PVSP can support various video compression algorithms and standards, such as MPEG-1, MPEG-2 H.263, and MPEG-4. Meanwhile, to improve the throughput of this codec system, some paralleling approaches are exploited on different levels, which include pipeline, tree adder, and SIMD (single instruction stream, multiple data streams). PVSP is estimated to have approximately 320k gates and it can accomplish MPEG-2 MP@ML encoding in real-time at the frequency of 133MHz.