Kilo-Instruction Processors: Overcoming the Memory Wall

Abstract

Historically, advances in integrated circuit technology have driven improvements in processor microarchitecture and led to today's microprocessors with sophisticated pipelines operating at very high clock frequencies. However, performance improvements achievable by high-frequency microprocessors have become seriously limited by main-memory access latencies because main-memory speeds have improved at a much slower pace than microprocessor speeds. It's crucial to deal with this performance disparity, commonly known as the memory wall, 1 to enable future high-frequency microprocessors to achieve their performance potential.To overcome the memory wall, we propose kilo-instruction processors-superscalar processors that can maintain a thousand or more simultaneous in-flight instructions. Doing so means designing key hardware structures so that the processor can satisfy the high resource requirements without significantly decreasing processor efficiency or increasing energy consumption. Nature of the memory wallOne of the first approaches to the memory wall problem was the development of cache memory hierarchies. Cache memories exploit program locality and can dramatically reduce the number of long-latency accesses to main memory. The first level, or L1 cache, is built into the processor core and typically takes one to three processor clock cycles to access. If there is a miss in the L1 cache, the on-chip L2 cache takes on the order of 10 processor cycles. Accessing main memory, on the other hand, takes at least an order of magnitude longer, and in the future this will become two orders of magnitude, that is, several hundred clock cycles. (In general, the cache hierarchy can have more than two levels, but to simplify our discussion here, we assume two levels with the understanding that the same principles apply to systems with deeper cache hierarchies.)Modern superscalar processors employ outof-order execution as a way of smoothing out disruptions caused by data cache misses (see the "Hiding latency in superscalar processors" sidebar). If a load instruction should experience a data cache miss, then instructions that depend on the miss data must wait in the issue queue(s). Meanwhile, independent instructions are free to execute; they issue from the issue queue(s) and essentially "pass" the blocked load instruction and its dependent instructions. For an L1 cache miss, these outof-order instructions can often completely hide the L2 access latency, so the miss causes little or no performance loss.This approach is much less effective for the long L2 cache misses, however. For example, along the top of Figure 1 is a sequence of instructions in program order. Following a

Diverge-Merge Processor (DMP): Dynamic Predicated Execution of Complex Control-Flow Graphs Based on Frequently Executed Paths

“…However, even with a perfect estimator, dual-path/multipath has less potential than DMP because (1) dual-path is applicable to one lowconfidence branch at a time (as explained previously in Section 5.1), (2) the overhead of dual-path/multipath is still much higher than that of DMP for a low-confidence branch because dual-path/multipath executes the same instructions twice/multiple times after a controlindependent point in the program. 13 These numbers are actually lower than what was previously published [17] because our baseline branch predictor uses a different algorithm and has a much higher prediction accuracy than that of [17]. Figure 15 (left) shows the average increase/reduction due to DMP in the number of fetched/executed instructions, maximum power, energy, and energy-delay product compared to the baseline.…”

“…Our baseline employs a 2KB enhanced JRS confidence estimator [19], which has 14% PVN ( accuracy) and 70% SPEC ( coverage) [17]. 13 Even with a 512-byte estimator, DMP still provides 18.4% performance improvement. The benefit of dual-path/multipath increases significantly with a perfect estimator because dual-path/multipath has very high overhead as shown in Figure 7, and a perfect confidence estimator eliminates the incurrence of this large overhead for correctly-predicted branches.…”

“…In the near future, processors are expected to support a large number of in-flight instructions [30,42,10,7,13] to extract both ILP and memory-level parallelism (MLP). As shown by previous research [27,40,41,30,42], the performance improvement provided by both pipelining and large instruction windows critically depends on the accuracy of the processor's branch predictor.…”

MLP-Aware Dynamic Instruction Window Resizing in Superscalar Processors for Adaptively Exploiting Available Parallelism

“…To the best of our knowledge, Cristal et al were the first to suggest that the ability to support many in-flight instructions is very effective in overcoming the memory wall [22]. Unfortunately, they thought (as did many others) that a simple implementation, based on enlarging the window resources, was impractical because of the delay, area, and power overheads.…”

Performance of Dynamic Instruction Window Resizing for a Given Power Budget under DVFS Control

“…If a last-level cache (LLC) miss occurs, the processor stalls waiting for the requested data to be retrieved from main memory, the latency of which is hundreds of clock cycles. To mitigate against this, aggressive out-of-order execution is an effective approach [1]. In this approach, the instruction window, composed of the reorder buffer (ROB), the issue queue (IQ), and the load/store queue (LSQ), is extensively enlarged and instructions are aggressively reordered at issue time.…”