The article proposes a new Kalman depth estimation and an improved swarm intelligence optimisation algorithm for adaptive tuning of servo gain for image-based visual servo control. First, a Kalman depth estimation model is established from the principle of image-based visual servoing, and two state equations are designed for depth estimation based on the number of state quantities. Second, the improved sparrow search algorithm is proposed to tune the servo gain adaptively to improve the convergence speed and stability. To verify the effectiveness of the proposed method, the conventional image-based visual servoing and conventional Kalman estimation are reproduced and compared with the proposed method, and the simulation is completed on the Simulink simulation platform for verification. Finally, the experiments are completed in the robotic arm experimental platform. Both the simulation and experimental results show the effectiveness of the proposed method, which reduces the redundancy of the camera and shortens the convergence time.