Group train operation (GTO) applications have reduced the tracking intervals for overloaded trains, and can affect the efficiency of rail transport. In this paper, we first analyze the differences between GTO and traditional operation (TO). A new mathematical model and simulated annealing algorithm are then used to study the problem of mixed group train operation. The optimization objective of this model is to maximize the transportation volume of special heavy-haul railway lines within the optimization period. The main constraint conditions are extracted from the maintenance time, the minimum ratio of freight volume, and the committed arrival time at each station. A simulated annealing algorithm is constructed to generate the mixed GTO plan. Through numerical experiments conducted on actual heavy-haul railway structures, we validate the effectiveness of the proposed model and meta-heuristic algorithm. The results of the first contrastive experiment show that the freight volume for group trains is 37.5% higher than that of traditional trains, and the second experiment shows a 30.6% reduction in the time during which the line is occupied by trains in GTO. These findings provide compelling evidence that GTO can effectively enhance the capacity and reduce the transportation time cost of special heavy-haul railway lines.