Efficient automatic scheduling of imaging and vision pipelines for the GPU

Abstract

We present a new algorithm to quickly generate high-performance GPU implementations of complex imaging and vision pipelines, directly from high-level Halide algorithm code. It is fully automatic, requiring no schedule templates or hand-optimized kernels. We address the scalability challenge of extending search-based automatic scheduling to map large real-world programs to the deep hierarchies of memory and parallelism on GPU architectures in reasonable compile time. We achieve this using (1) a two-phase search algorithm that first ‘freezes’ decisions for the lowest cost sections of a program, allowing relatively more time to be spent on the important stages, (2) a hierarchical sampling strategy that groups schedules based on their structural similarity, then samples representatives to be evaluated, allowing us to explore a large space with few samples, and (3) memoization of repeated partial schedules, amortizing their cost over all their occurrences. We guide the process with an efficient cost model combining machine learning, program analysis, and GPU architecture knowledge. We evaluate our method’s performance on a diverse suite of real-world imaging and vision pipelines. Our scalability optimizations lead to average compile time speedups of 49x (up to 530x). We find schedules that are on average 1.7x faster than existing automatic solutions (up to 5x), and competitive with what the best human experts were able to achieve in an active effort to beat our automatic results.

Shisha: Online scheduling of CNN pipelines on heterogeneous architectures

“…In order to partition and schedule pipelines, current approaches rely on designing cost models to steer design space exploration [1,3]. For instance, the auto-scheduler in [1] explores over ten thousand schedules for a single CNNlayer Halide [24] pipelines.…”

“…Sophisticated cost models, some of them using ML-models themselves, have been proposed and used in [1,2,16,20,33,35,38]. These models, however, require extensive training for near-optimal solutions [3], are sensitive to changes in the execution environment (e.g., DVFS) and architectural parameters, need in-depth architectural knowledge for model updates, and do not consider the impact of heterogeneous or chiplet architectures. As heterogeneity at different levels of processing (e.g.…”

“…The configuration consists of the number of pipeline stages, CNN layers per pipeline stage and a mapping of pipeline stages to EPs. In the literature, various stochastic optimization and machine learning algorithms have been used such as Simulated Annealing [38], evolutionary algorithms [1,30], reinforcement learning [2,23] and deep neural network techniques [3]. The design space under consideration is large and complex, requiring tens of thousands of trials in order to reach a near optimum with current search schemes.…”

An Asymptotic Cost Model for Autoscheduling Sparse Tensor Programs

“…Transformations like loop reordering or loop fusion can have large effects on constant factors in the runtime of dense tensor programs [3]. However, the same transformations can have even larger asymptotic effects when tensors are sparse.…”

“…The user might need to schedule too many kernels, or may be unfamiliar with the intricacies of the tensor compiler in question. Sparse systems must follow the lead of dense systems, which are now moving towards automatic scheduling [2,3,27]. Automatic scheduling promises a realistic path towards integration into high-level systems like SciPy [38] or TensorFlow [1].…”