The advancement of smart agriculture, with information technology serving as a pivotal enabling factor, plays a crucial role in achieving food security, optimizing production efficiency, and preserving the environment. Simultaneously, wireless communication technology holds a critical function within the context of applying the Internet of Things in agriculture. In this research endeavor, we present an algorithm for lightweight channel authentication based on frequency-domain feature extraction. This algorithm aims to distinguish between authentic transmitters and unauthorized ones in the wireless communication context of a representative agricultural setting. To accomplish this, we compiled a dataset comprising legitimate and illegitimate communication channels observed in both indoor and outdoor scenarios, which are typical in the context of smart agriculture. Leveraging its exceptional perceptual capabilities and advantages in parallel computing, the Transformer has injected fresh vitality into the realm of signal processing. Consequently, we opted for the lightweight MobileViT as our foundational model and designed a frequency-domain feature extraction module to augment MobileViT’s capabilities in signal processing. During the validation phase, we conducted a side-by-side comparison with currently outstanding ViT models in terms of convergence speed, precision, and performance parameters. Our model emerged as the frontrunner across all aspects, with FDFE-MobileViT achieving precision, recall, and F-score rates of 96.6%, 95.6%, and 96.1%, respectively. Additionally, the model maintains a compact size of 4.04 MB. Through comprehensive experiments, our proposed method was rigorously verified as a lighter, more efficient, and more accurate solution.