Large‐scale usage of agrochemicals in the soils found in the highland regions of Western Ghats, Kerala, India, is often equated with soil contamination and land degradation. Kerala, being the second largest cardamom producer in the world, unscrupulously uses agrochemicals in cardamom plantations, which release high levels of potentially toxic elements (PTEs) into the soils. Here, the PTEs from 25 agricultural soils were taken from high‐range mountain landscapes of southern Western Ghats, India, to evaluate the sources and levels of soil pollution. The data revealed that the soil samples from the study area were contaminated with PTEs, and their observed average concentration was in the order of Ba > Zr > V > Cr > Ni > Sr > Zn > Cu. Geochemical maps were generated to understand the hotspots in trace element contamination. Correlation matrix and factor analysis were applied to apportion the sources of PTEs. Enrichment factor and geochemical accumulation index indicated slight enrichment in all the elements analyzed. The contamination factor varied from 15.41 to 25.58, signifying a considerable degree of contamination in the area with respect to soil pollution. The pollution load index (PLI) values ranged from 0.97 to 1.99, indicating progressive deterioration of the sites. High degree of contamination and PLI values implied that not only geogenic sources, but also intensified anthropogenic activities may contribute to trace element accumulation. This work provides insights on the accumulation of PTEs in the soils, which will pose a great health risk to the local population through the consumption of various agricultural products cultivated in the area.