In this study, the water quality of the irrigation system and concentration of trace metals in the sediments were combined to establish a farmland trace metal contamination and management model (FTM_CMM). The purpose of this model was to clarify the main sources of the trace metals that have caused the contamination of paddy soil in central Taiwan. The results of the model simulation showed that the trace metals in the paddy soil mainly came from the irrigation water and especially from the sediments in the irrigation channels. The contribution of the sediments in the irrigation channel to the individual trace metals in the paddy soil ranged from 56% to 72% as the contributions for Cr, Cu, Ni, and Zn were 72%, 68%, 56%, and 62%, respectively. The trace metal species according to their concentration in the contaminated soil ranked in the order of Zn > Cr ≈ Cu > Ni, which is about the same as in the channel sediment. During the simulation process, Cr, Cu, Ni, and Zn exceeded the control standards for farmlands in the 18th, 12th, 13th, and 17th years, respectively. This highlights that, in addition to the management of irrigation water quality, the management of trace metal contaminated sediment in adjacent canal irrigation systems is also an important part of the prevention of trace metal contamination in farmland.