Because of the iterative process in the rate control algorithm of the H.264 draft, it is not suitable for real-time transmission. A two-layer real-time rate control algorithm based on the Lagrange optimum algorithm is proposed. This algorithm consists of two layers, namely, the group of pictures (GOP) layer and the frame layer. The simulation results show that the proposed algorithm is competitive in terms of peak signal-to-noise ratio (PSNR), real bitrates and the Q-value among frames over the original algorithm.H.264 is the latest video coding standard proposed by the Video Coding Experts Group (VCEG) and the Moving Picture Experts Group (MPEG) [1] . Its rate control algorithm named after the H.264 test model 5(TM5) is developed from test model 5 of MPEG2. TM5 consists of four steps [2] , namely, allocating propel bits to each frame and initiating the quantization parameters; exploiting rate-distortion(R-D) model to choose the encoding pattern and estimate motion; updating Q value according to buffer's states and encoding bits; using R-D model to choose new motion vector (MV) and Macro block's encoding pattern.The quantization parameter Q used by the H.264 rate control algorithm and R-D model introduces the emulation problem [3] below: when performing R-D optimization to frame, the sum of absolute differences (SAD) of the current frame must be calculated first to choose the quantization parameter; while SAD can be only obtained after the R-D optimizing process. Because of this disadvantage of the H.264 draft, it is not suitable for real-time transmission. We propose a new two-layer real-time rate control algorithm in this paper to address this problem.