In recent years, fatigue has become an important issue in modern life that cannot be ignored, especially in some special occupations. Agricultural workers are high-risk occupations that, under fatigue conditions over a long period, will cause health problems. In China, since very few studies have focused on the fatigue state of agricultural workers, we were interested in using electrocardiogram (ECG) signals to analyze the fatigue state of agricultural workers. Healthy agricultural workers were randomly recruited from hilly orchards in South China. Through the field experiment, 130 groups of 5-min interval ECG signals were collected, and we analyzed the ECG signal by HRV. The time domain (meanHR, meanRR, SDNN, RMSSD, SDSD, PNN20, PNN50 and CV), frequency domain (VLF percent, LF percent, HF percent, LF norm, HF norm and LF/HF) and nonlinear parameters (SD1, SD2, SD1/SD2 and sample entropy) were calculated and Spearman correlation coefficient analysis and Mann–Whitney U tests were performed on each parameter for further analysis. For all subjects, nine parameters were slightly correlated in nonfatigue and fatigue state. Six parameters were significantly increased and ten HRV parameters were significantly decreased compared the nonfatigue state. As for males, fifteen parameters were significantly different, and for females, eighteen parameters were significantly different. In addition, the probability density functions of SDNN, SDSD, VLF%, HFnorm and LF/HF were significantly different in nonfatigue and fatigue state for different genders, and the nonlinear parameters become more discrete compared the nonfatigue state. Finally, we obtained the most suitable parameters, which reflect the fatigue characteristics of orchard workers under different genders. The results have instructional significance for identifying fatigue in orchard workers and provide a convincing and valid reference for clinical diagnosis.