Fixed point chaos based stream cipher (FPC-SC) is presented in this paper to encrypt the speech signal. Fixed point chaos based pseudo random bit generator (FPC-PRBG) is used as key sequence in stream cipher system. Two chaotic Lorenz and Chen systems are combined to generate Gold FPC-PRBG. The results show that FPC-PRBG has good statistical randomness and encryption performance measure and can be used to produce high speech security level. FPC-PRBG is implemented using Xilinx system generator (XSG) with Xilinx Virtex 4 FPGA device, the system is accessed routed in this device with throughputs 818.64,7978.88 and 780.24 Mbits/sec for Lorenz, Chen and Gold FPC-PRBG respectively, where 40 bits fixed point operation are used.