This paper investigates the problem of multi-robot formation control strategies in environments with obstacles based on deep reinforcement learning methods. To solve the problem of value function overestimation in the deep deterministic policy gradient (DDPG) algorithm, this paper proposes an improved multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm under the CTDE framework combined with the twin delayed deep deterministic policy gradient (TD3) algorithm, which adopts a prioritized experience replay strategy to improve the learning efficiency. For the problem of difficult obstacle avoidance for a robot formation, a hybrid reward mechanism is designed to use different formation maintenance strategies in obstacle areas and obstacle-free areas to achieve the control goal of obstacle avoidance by reasonably changing the formation. The simulation experiments verified the effectiveness of the multi-robot formation control strategy designed in this paper, and comparative simulations verified that the algorithm has a faster convergence speed and more stable performance.