With rapidly developing autonomous vehicle (AV) technologies, the optimal driving strategy should consider multi-objective optimization problems of large-scale transportation systems, including safety and efficiency. Different driving strategies have different performance, and there is an interaction between vehicles with different strategies. Since the Nash equilibrium is hard to find for an n-player game, it is difficult to get an analytical solution to this multi-objective optimization problem. Therefor a coevolutionary algorithm is proposed to explore the interactions between populations with different strategies and investigate the cooperation and competition among vehicles. Combining the multi-objective optimization algorithm and the incentive mechanism of survival of the fittest reproduction law, the system structure reaches a stable equilibrium state and the optimal group driving strategy evolves. Simulation results, with 40,000 vehicles driving in Luxembourg SUMO Traffic Scenario, demonstrate that the rational-rational strategy performs best among six typical strategies. Meanwhile, the accident rate drops by 56%, while the overall average speed increases by 30%. The results of multi-vehicle and multi-objective coevolution are enlightening in designing optimal driving strategy with AVs.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.