Swarm of drones, as an intensely significant category of swarm robots, is widely used in various fields, e.g., search and rescue, detection missions, military, etc. Because of the limitation of computing resource of drones, dealing with computation-intensive tasks locally is difficult. Hence, the cloud-based computation offloading is widely adopted, nevertheless, for some latency-sensitive tasks, e.g., object recognition, path planning, etc., the cloud-based manner is inappropriate due to the excessive delay. Even in some harsh environments, e.g., disaster area, battlefield, etc., there is no wireless infrastructure existed to combine the drones and cloud center. Thus, to solve the problem encountered by cloud-based computation offloading, in this paper, Fog Computing aided Swarm of Drones (FCSD) architecture is proposed. Considering the uncertainty factors in harsh environments which may threaten the success of FCSD processing tasks, not only the latency model, but also the reliability model of FCSD is constructed to guarantee the high reliability of task completion. Moreover, in view of the limited battery life of the drone, we formulated the problem as the task allocation problem which minimized the energy consumption of FCSD under the constraints of latency and reliability. Furthermore, to speed up the process of the optimization problem solving to improve the practicality, relying on the recent advances in distributed convex optimization, we develop a fast Proximal Jacobi Alternating Direction Method of Multipliers (ADMM) based distributed algorithm. Finally, simulation results validate the effectiveness of our proposed scheme. INDEX TERMS Swarm of drones, distributed fog computing, latency, reliability, energy consumption.