On the Use of Commodity Ethernet Technology in Exascale HPC Systems

Non‐minimal adaptive routing based on explicit congestion notifications

“…Per input-buffer flow control is supported by Priority-based Flow Control (PFC), which supports individual pauses per buffer but does not handle buffer credits. 7 The conditional decision in that original proposal relied on the PAUSE status of each output port, but such implementation presents poor throughput under uniform traffic at high loads and poor fairness and average latency under adversarial traffic, as shown in an earlier work. The application of this Ethernet technology to build Dragonfly networks follows the ideas introduced in an earlier work, 7 including hierarchical MAC address rewriting and TCAM compaction, which are required for the mechanism to work but are not relevant for this study.…”

“…7 The conditional decision in that original proposal relied on the PAUSE status of each output port, but such implementation presents poor throughput under uniform traffic at high loads and poor fairness and average latency under adversarial traffic, as shown in an earlier work. 7 Instead of pauses, our proposal characterizes each output port with a percentage value, which defines the probability of minimal routing through that output. This design was proposed and partially evaluated in our previous work 9 and will be presented in more detail in Section 3.…”

“…The application of this Ethernet technology to build Dragonfly networks follows the ideas introduced in an earlier work, 7 including hierarchical MAC address rewriting and TCAM compaction, which are required for the mechanism to work but are not relevant for this study. Per input-buffer flow control is supported by Priority-based Flow Control (PFC), which supports individual pauses per buffer but does not handle buffer credits.…”

“…Routing is implemented using a flow forwarding table that contains multiple flow entries, which are instantiated proactively. 7 Instead of pauses, our proposal characterizes each output port with a percentage value, which defines the probability of minimal routing through that output. Deadlock avoidance in the Dragonfly topology is guaranteed by leveraging 802.1p classes of service to avoid cyclic dependencies, resembling the ordering in the work of Kim et al 1 Adaptive routing is implemented using conditional OpenFlow rules, first proposed for non-minimal adaptive routing in an earlier work.…”

“…presents a competitive latency and a sufficiently quick response time, according to our previous analysis of HPC applications requirements in an earlier work 7. …”