Soil pollution by potentially toxic elements (PTEs) has become a core issue around the world. Knowledge of the spatial distribution of PTEs in soil is crucial for soil remediation. Portable X-ray fluorescence spectroscopy (p-XRF) provides a cost-saving alternative to the traditional laboratory analysis of soil PTEs. In this study, we collected 293 soil samples from Fuyang County in Southeast China. Subsequently, we used several geostatistical methods, such as inverse distance weighting (IDW), ordinary kriging (OK), and empirical Bayesian kriging (EBK), to estimate the spatial variability of soil PTEs measured by the laboratory and p-XRF methods. The final maps of soil PTEs were outputted by the model averaging method, which combines multiple maps previously created by IDW, OK, and EBK, using both lab and p-XRF data. The study results revealed that the mean PTE content measured by the laboratory methods was as follows: Zn (127.43 mg kg−1) > Cu (31.34 mg kg−1) > Ni (20.79 mg kg−1) > As (10.65 mg kg−1) > Cd (0.33 mg kg−1). p-XRF measurements showed a spatial prediction accuracy of soil PTEs similar to that of laboratory analysis measurements. The spatial prediction accuracy of different PTEs outputted by the model averaging method was as follows: Zn (R2 = 0.71) > Cd (R2 = 0.68) > Ni (R2 = 0.67) > Cu (R2 = 0.62) > As (R2 = 0.50). The prediction accuracy of the model averaging method for five PTEs studied herein was improved compared with that of the laboratory and p-XRF methods, which utilized individual geostatistical methods (e.g., IDW, OK, EBK). Our results proved that p-XRF was a reliable alternative to the traditional laboratory analysis methods for mapping soil PTEs. The model averaging approach improved the prediction accuracy of the soil PTE spatial distribution and reduced the time and cost of monitoring and mapping PTE soil contamination.