dynamicMF: A Matrix Factorization Approach to Monitor Resource Usage in High Performance Computing Systems

Abstract

High performance computing (HPC) facilities consist of a large number of interconnected computing units (or nodes) that execute highly complex scientific simulations to support scientific research. Monitoring such facilities, in real-time, is essential to ensure that the system operates at peak efficiency. Such systems are typically monitored using a variety of measurement and log data which capture the state of the various components within the system at regular intervals of time. As modern HPC systems grow in capacity and complexity, the data produced by current resource monitoring tools is at a scale that it is no longer feasible to be visually monitored by analysts. We propose a method that transforms the multi-dimensional output of resource monitoring tools to a low dimensional representation that facilitates the understanding of the behavior of a High Performance Computing (HPC) system. The proposed method automatically extracts the low-dimensional signal in the data which can be used to track the system ef-ficiency and identify performance anomalies. The method models the resource usage data as a three dimensional tensor (capturing resource usage of all compute nodes for difference resources over time). A dynamic matrix factorization algorithm, called dynamicMF, is proposed to extract a low-dimensional temporal signal for each node, which is subsequently fed into an anomaly detector. Results on resource usage data collected from the Lonestar 4 system at the Texas Advanced Computing Center show that the identified anomalies are correlated with actual anomalous events reported in the system log messages.

A novel virtual-communicated evolution learning recommendation

“…Matrix factorization (MF) refers to the decomposition of potential characteristics of users and items embedded in the relationship matrix. Recently, several studies (Gemulla et al, 2011;Huang et al, 2018;Koren et al, 2009;Lee and Seung, 2000;Luo et al, 2014;Rendle et al, 2020;Salakhutdinov and Mnih, 2007;Sorkunlu et al, 2018;Wu et al, 2020;Yuan et al, 2021;Zeng et al, 2015;Zhang et al, 2021) have discussed the effectively decomposing process on the given matrix. Koren et al (2009) described the characteristics of the users and items with two decomposed low-dimensional matrices, and then inner-product the two derived matrices to reconstruct for rating prediction.…”

“…RSNMF (Luo et al ., 2014) regularizes a single element on the non-negative updated process which depends on each feature vector instead of on the whole matrix. DynamicMF (Sorkunlu et al ., 2018) automatically captures low-dimensional features to efficiently enhance the factorization process which could be applied on several real domains. SDMF (Wu et al ., 2020) extracts the latent characteristics by MF and derives the corresponding binary code for the geometrical structures collectively embedded in users and items learned from vector space.…”

“…However, deriving user preference is usually a difficult task, since even users themselves could not accurately articulate their real interests. In this circumstance, matrix factorization (MF) techniques (Gemulla et al, 2011;Huang et al, 2018;Immer et al, 2020;Kawale et al, 2015;Koren et al, 2009;Lee and Seung, 2000;Luo et al, 2014;Park et al, 2017;Rendle et al, 2020;Salakhutdinov and Mnih, 2007;Sorkunlu et al, 2018;Tran et al, 2018;Trigeorgis et al, 2017;Wang and Ma, 2020;Wu et al, 2020Wu et al, , 2021aYuan et al, 2021;Zeng et al, 2015;Zhang and Chow, 2016;Zhang et al, 2021) have been widely applied to recommendation systems to discover latent characteristics from explicit user feedback which implicitly describes the features of users and items. MF decomposes the user rating matrix on items into two separate matrices in a shared latent space; one matrix depicts the user vectors and the other depicts the item vectors.…”