-Codebook design plays an important role in the performance of signal processing based on vector quantization (VQ) just as speech coding, data compression and pattern recognition. LBG is one of the most effective algorithms and is widely used in codebook generation. Some problems still exist while its performance is remarkable. The LBG algorithm is easy to fall into local optimum. Usually there is strong correlation between the best solution and the initial selection for codebook design. It means that the quantization performance of codebooks from the same training data may varies in a certain range. There is also a certain probability for the algorithm to generate empty voronoi cell. In order to solve these problems, a novel algorithm based on ant colony clustering algorithm and genetic algorithm is proposed in this paper. The new algorithm takes advantage of the excellent global optimal searching ability of genetic algorithm. At the same time, the ant colony clustering algorithm is combined into the process. The dynamic change of the searching direction is adopted during crossover stage. The simulation results of line spectrum frequency parameters in mixed linear excitation prediction (MELP) show that the proposed algorithm is more efficient in its quantization performance compared to that of the LBG and genetic algorithms. Meanwhile, it has good stability in quantization performance.