In this work, the influence of the intermetallic particle fragmentation during hot rolling of the as cast structure on the evolution of textures in aluminum alloys 8011, 5182 and 1565 was investigated. For this purpose, laboratory multi-pass rolling of the cast material was carried out. At various degrees of hot rolling deformation, the process was stopped, and the metal was quenched and sent for optical and electron microscopy to investigate the large intermetallic particles. In addition, the grain structure was studied and an X-ray analysis was carried out in order to determine the main texture components. Some of the samples were held at a temperature above the recrystallization threshold and then cooled in air; the grain structure and texture composition were also studied. In addition, the simulation of the texture evolution was carried out under various modes of rolling of aluminum alloys, taking into account the process of fragmentation of intermetallic particles. The investigation showed that intermetallic compounds with a deformation degree of 1.8, on average, decrease the particle size by 5–7 times. The large eutectic particles remaining after homogenization are drawn out in the direction of deformation and are crushed, increasing their number accordingly. Therefore, the most favorable stage for the formation of recrystallization nuclei on particles is the moment when they are already numerous and their sizes are much larger than subgrains. Simulation of hot rolling of the investigated alloys showed that considering the factor of fragmentation of intermetallic particles during hot deformation of the as-cast structure significantly increases the accuracy of the results.