An optimized tensor completion library for multiple GPUs

Abstract

Tensor computations are gaining wide adoption in big data analysis and artificial intelligence. Among them, tensor completion is used to predict the missing or unobserved value in tensors. The decomposition-based tensor completion algorithms have attracted significant research attention since they exhibit better parallelization and scalability. However, existing optimization techniques for tensor completion cannot sustain the increasing demand for applying tensor completion on ever larger tensor data. To address the above limitations, we develop the first tensor completion library cuTC on multiple Graphics Processing Units (GPUs) with three widely used optimization algorithms such as alternating least squares (ALS), stochastic gradient descent (SGD) and coordinate descent (CCD+). We propose a novel TB-COO format that leverages warp shuffle and shared memory on GPU to enable efficient reduction. In addition, we adopt the auto-tuning method to determine the optimal parameters for better convergence and performance. We compare cuTC with state-of-the-art tensor completion libraries on real-world datasets, and the results show cuTC achieves significant speedup with similar or even better accuracy. CCS CONCEPTS• Computer systems organization → Parallel architectures; • Computing methodologies → Parallel algorithms.

Efficient, Out-of-Memory Sparse MTTKRP on Massively Parallel Architectures

“…Optimizing sparse MTTKRP has been the subject of several prior studies, which propose sparse tensor formats along with parallel algorithms to process/analyze the data. List-based formats, such as F-COO [30], GenTen [39], and TB-COO [12], explicitly store the multi-dimensional coordinates of each non-zero element. To reduce atomic operations, these formats store multiple mode-specific copies of the tensor and/or extra scheduling information, which substantially increases their memory footprint.…”

“…Segmented scan and reduction [42,52] have been used to reduce the synchronization cost of sparse workloads [6,12,29,51,53]. Prior studies apply these primitives to mode-specific formats with delineated and/or sorted groups of non-zero elements according to the target mode.…”

“…The state-of-the-art tensor formats for massively parallel GPUs use list-based [12,29,39] or tree-based [34,36] data structures to store high-dimensional sparse data in a modespecific form. In this section, we provide an overview of the flagged coordinate (F-COO) and MM-CSF formats, which are representative of the two main format categories for GPU architectures.…”

“…TD algorithms for high-dimensional sparse data are challenging to execute on emerging parallel architectures due to their low arithmetic intensity, irregular memory access, workload imbalance, and synchronization overhead [10,17]. To improve the performance of these memory-bound workloads, recent studies [12,29,34,36,39] exploit massively parallel architectures equipped with High Bandwidth Memory (HBM), namely GPUs, to accelerate the MTTKRP kernel. While such accelerators deliver memory bandwidth exceeding 2 TB/s [1], they suffer from limited memory capacity and high memoryaccess latency, which is in the order of hundreds of processor cycles [20,32].…”

“…However, these strategies result in formats that are mode-specific, where nonzero elements are organized/accessed according to a specific mode (i.e., dimension) orientation. Mode-specific formats typically require keeping multiple tensor copies [12,29,36] and/or extra mapping and scheduling information (e.g., flags or arrays) about groups of data-dependent non-zero elements for every mode [12,29,39], which can significantly increase their overall memory footprint.…”