Minimizing the offloading latency of agricultural drip irrigation and fertilization tasks has long been a pressing issue in agricultural drip irrigation and fertilization wireless sensor networks (AIFWSNs). The introduction of edge computing as a robust and practical aid to cloud computing in AIFWSNs can significantly improve the execution speed of agricultural drip irrigation and fertilization tasks and effectively reduce the task offloading latency. Therefore, this paper investigates the optimization method of drip irrigation and fertilization task offloading delay in AIFWSNs based on edge computing and proposes a new edge task offloading method for AIFWSNs, namely, Quantum Chaotic Genetic Optimization Algorithm (QCGA). This paper introduces a novel quantum operator in QCGA, comprising a quantum non-gate and a quantum rotation gate, to improve the algorithm’s global search capability. The quantum operator accomplishes the updating of quantum rotating gates without querying the quantum rotation angle table, which reduces the computational complexity of introducing quantum optimization into the task offloading problem of AIFWSNs. This paper proposes a new chaotic operator to make the initial solution more uniformly distributed in the search space by chaotic mapping. This paper’s simulation experiments compared QCGA and snake optimizer (SO), genetic algorithm (GA), particle swarm optimization (PSO), sequential offloading, and random offloading methods. Simulation results showed that, compared with SO, GA, PSO, sequential offloading, and random offloading methods, the average delay of QCGA was reduced by 9.96%, 26.78%, 29.31%, 44.67%, and 61.24%.