Microscopic content segregation is among the important reasons for the anisotropy of mechanical properties in the cast-rolled sheets of the 7B05 aluminum alloy. It is of great significance to study the uniformity of aluminum alloys in terms of the microscopic composition and structure. In this study, a precise method for composition quantification based on micro-beam X-ray fluorescence spectroscopy is established by parameter optimization and a calibration coefficient. Furthermore, this method was applied for exploring and quantifying the relationship between recrystallization and deformation microstructures. The results show that the comprehensive measurement effects of all elements are the best when the X-ray tube voltage is 50 kV, the current is 150 μA, and the single-pixel scanning time is 100 ms. After verification, the sum of differences between the measured values and the standard values for all elements using the calibration coefficient is only 0.107%, which confirms the accuracy of the optimized quantitative method. Three types of segregation indexes in national standards were used to capture small differences, and finally ensure that the segregation degrees of elements are Ti > Fe > Cr > Cu > Mn > Zr > Zn > Al. The quantitative segregation results obtained by the spatial-mapping method show that the difference in the content of Al and Zn is approximately 0.2% between the recrystallization region and the deformation region, the difference in the content of Fe and Ti is 0.018% and 0.013%, the difference in the content of Cr, Cu and Zr is approximately 0.01%, and the difference in the content of Mn is not obvious, only 0.004%.