A Deep Reinforcement Learning Approach for Global Routing

Abstract

Global routing has been a historically challenging problem in electronic circuit design, where the challenge is to connect a large and arbitrary number of circuit components with wires without violating the design rules for the printed circuit boards or integrated circuits. Similar routing problems also exist in the design of complex hydraulic systems, pipe systems and logistic networks. Existing solutions typically consist of greedy algorithms and hard-coded heuristics. As such, existing approaches suffer from a lack of model flexibility and non-optimum solutions. As an alternative approach, this work presents a deep reinforcement learning method for solving the global routing problem in a simulated environment. At the heart of the proposed method is deep reinforcement learning that enables an agent to produce an optimal policy for routing based on the variety of problems it is presented with leveraging the conjoint optimization mechanism of deep reinforcement learning. Conjoint optimization mechanism is explained and demonstrated in details; the best network structure and the parameters of the learned model are explored. Based on the fine-tuned model, routing solutions and rewards are presented and analyzed. The results indicate that the approach can outperform the benchmark method of a sequential A* method, suggesting a promising potential for deep reinforcement learning for global routing and other routing or path planning problems in general. Another major contribution of this work is the development of a global routing problem sets generator with the ability to generate parameterized global routing problem sets with different size and constraints, enabling evaluation of different routing algorithms and the generation of training datasets for future data-driven routing approaches.

Deep Q-Learning Based Optimization of VLC Systems With Dynamic Time-Division Multiplexing

“…Liao et al applied the DQL to solve the 3D global routing problem [6]. Their proposed approach first decomposes the multiple pin nets into two pin subnets and then routes the subnets on a weighted 3D mesh.…”

“…The first case in the above equation shows that the application of action a 0 on any state, does not change the duration of time-slots, and hence the state remains unchanged (i.e., s t+1 = s t ). The second case is for any action a t = a k , where (k > 0) applied on the state s t , but the action a t has a problem that it reduces the duration of at least one timeslot τ i ∈ s t whose existing value is τ i < τ min + δ, i.e., any decrease in the τ i value is the violation of the constraint (6). In both these cases, the environment returns a reward but does not change its state.…”

“…They played better than humans in some cases [5]. Some recent applications of Q-learning include: (i) autonomous control of a power network; (ii) autonomous determination of congestion-free paths in global routing [6]; (iii) dynamic path planning [7]; and, (iv) path planning in grid graphs [8].…”

A Survey on Steiner Tree Construction and Global Routing for VLSI Design

“…MGR [115] is a multilevel 3D router that runs much faster than traditional 3D router [14]. Besides, for the last two years, researchers have tried to solve the GR problem by using machine learning-based approaches [116]- [118].…”

A Nesterov's accelerated quasi-Newton method for global routing using deep reinforcement learning

“…Thus deep reinforcement learning (DRL) is a potential approach to such applications. Recently, a reinforcement learning approach to global routing that uses first order gradient based method for training the DQN was proposed in [14]. This paper attempts to accelerate the training of deep Q-networks by introducing a second order Nesterov's accelerated quasi-Newton method to get better routing solutions in fewer episodes.…”