Due to the rapid population growth and over-application of fertilizers in the Caohai surrounding farmlands, controlling the non-point source pollution in the Caohai Lakeside Zone is significant for the local ecology balance and human health safety. A total of 54 soil and 24 irrigation water samples were collected in the Caohai Lakeside Zone to evaluate the current status of soil fertility and heavy metal pollution by measuring soil and water heavy metal content and soil physicochemical properties such as soil total nitrogen, total phosphorus, organic matter, and soil pH. These results showed that the total amounts of organic matter, nitrogen, phosphorus, potassium, and effective nutrient content in the Caohai Lakeside Zone were all at a rich level according to Chinese soil nutrient grade standard; the content of lead (Pb), chromium (Cr), cadmium (Cd), mercury (Hg), and arsenic (As) in the soil all exceeds the safety standard of Chinese classification of soil environmental quality assessment. In addition, the over-standard rates of Cd and Cr were 24.1% and 14.8%, respectively. On the whole, the comprehensive quality index (CQI) of the soil in the Caohai Lakeside Zone is 6.48, which is attributed to the heavy Cd pollution and the good soil fertility. The heavy metal content of the irrigation water met the requirements of irrigation water quality. It is feasible to use the irrigation water of Caohai Lake in the actual agricultural production process. Therefore, in terms of the comprehensive management and pollution control of Caohai, special attention should be paid to the control of Cd, Pb, As, and Hg pollution sources.