For the application of the standard genetic algorithm in illustration art design, there are still problems such as low search efficiency and high complexity. This paper proposes an illustration art design model based on operator and clustering optimization genetic algorithm. First, during the operation of the genetic algorithm, the values of the crossover probability and the mutation probability are dynamically adjusted according to the characteristics of the population to improve the search efficiency of the algorithm, then the k-medoids algorithm is introduced to optimize the clustering of the genetic algorithm, and a cost function is used to carry out and evaluate the quality of clustering to optimize the complexity of the original algorithm. In addition, a multiobjective optimization genetic algorithm with complex constraints based on group classification is proposed. This algorithm focuses on the problem of group diversity and uses k-means cluster analysis operation to solve the problem of group diversity. The algorithm divides the entire group into four subgroups and assigns appropriate fitness values to reflect the optimal preservation strategy. A large number of computer simulation calculations show that the algorithm can obtain a widely distributed and uniform Pareto optimal solution, the evolution speed is fast, usually only a few iterations can achieve a good optimization effect, and finally the improved genetic algorithm is used to design the random illustration art. The example simulation shows that the improved algorithm proposed in this paper can achieve higher artistic and innovative illustration art design.