Holonomic quantum computing offers a promising paradigm for quantum computation due to its error resistance and the ability to perform universal quantum computations. Here, we propose a scheme for the rapid implementation of a holonomic swap gate in neutral atomic systems, based on the selective Rydberg pumping mechanism. By employing time-dependent soft control, we effectively mitigate the impact of off-resonant terms even at higher driving intensities compared to time-independent driving. This approach accelerates the synthesis of logic gates and passively reduces the decoherence effects. Furthermore, by introducing an additional atom and applying the appropriate driving field, our scheme can be directly extended to implement a three-qubit controlled-swap gate. This advancement makes it a valuable tool for quantum state preparation, quantum switches, and a variational quantum algorithm in neutral atom systems.