Text Level Graph Neural Network for Text Classification

Huang, Lianzhe; Ma, Dehong; Li, Sujian; Zhang, Xiaodong; Wang, Houfeng

doi:10.48550/arxiv.1910.02356

Cited by 32 publications

(36 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We discuss related works on the theory and applications of GNNs. There exist general GNN surveys [48], [49], [58], [100], [223], [241], [244], [251], works on theoretical aspects (spatial-spectral dichotomy [11], [63], the expressive power of GNNs [178], or heterogeneous graphs [224], [229]), analyzes of GNNs for specific applications (knowl-edge graph completion [5], traffic forecasting [119], [195], symbolic computing [137], recommender systems [220], text classification [115], or action recognition [3]), explainability of GNNs [232], and on software (SW) and hardware (HW) accelerators and SW/HW co-design [1]. We complement these works as we focus on parallelism and distribution of GNN workloads.…”

Section: Complementary Analysesmentioning

confidence: 99%

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis

Besta¹,

Hoefler²

2022

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNNs) are among the most powerful tools in deep learning. They routinely solve complex problems on unstructured networks, such as node classification, graph classification, or link prediction, with high accuracy. However, both inference and training of GNNs are complex, and they uniquely combine the features of irregular graph processing with dense and regular computations. This complexity makes it very challenging to execute GNNs efficiently on modern massively parallel architectures. To alleviate this, we first design a taxonomy of parallelism in GNNs, considering data and model parallelism, and different forms of pipelining. Then, we use this taxonomy to investigate the amount of parallelism in numerous GNN models, GNN-driven machine learning tasks, software frameworks, or hardware accelerators. We use the work-depth model, and we also assess communication volume and synchronization. We specifically focus on the sparsity/density of the associated tensors, in order to understand how to effectively apply techniques such as vectorization. We also formally analyze GNN pipelining, and we generalize the established Message-Passing class of GNN models to cover arbitrary pipeline depths, facilitating future optimizations. Finally, we investigate different forms of asynchronicity, navigating the path for future asynchronous parallel GNN pipelines. The outcomes of our analysis are synthesized in a set of insights that help to maximize GNN performance, and a comprehensive list of challenges and opportunities for further research into efficient GNN computations. Our work will help to advance the design of future GNNs.

show abstract

Section: Complementary Analysesmentioning

confidence: 99%

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis

Besta¹,

Hoefler²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Phan et al [24] is the first work that has attempted to apply Graph Neural Networks classification for predicting the range of story points of the Deep-SE dataset. They rely on the graph construction and training process of the TextLevelGNN engine [14] to solve the problem of estimation. Although TextLevelGNN is a well-known homogeneous GNN model in NLP research, Phan et al [24] show that the original GNN models can cause negative impacts on estimating story points due to the large size of vocabulary and number of edges for graph construction.…”

Section: Rq4: Can Heterosp's Graphs Support Homogeneous Gnn In Effort...mentioning

confidence: 99%

Heterogeneous Graph Neural Networks for Software Effort Estimation

Phan,

Jannesari

2022

Preprint

View full text Add to dashboard Cite

Background. Software effort can be measured by story point [35]. Story point estimation is important in software projects' planning. Current approaches for automatically estimating story points focus on applying pre-trained embedding models and deep learning for text regression to solve this problem. These approaches require expensive embedding models and confront challenges that the sequence of text might not be an efficient representation for software issues which can be the combination of text and code.Aims. We propose HeteroSP, a tool for estimating story points from textual input of Agile software project issues. We select GPT2SP[12] and Deep-SE [8] as the baselines for comparison.Method. First, from the analysis of the story point dataset [8], we conclude that software issues are actually a mixture of natural language sentences with quoted code snippets and have problems related to large-size vocabulary. Second, we provide a module to normalize the input text including words and code tokens of the software issues. Third, we design an algorithm to convert an input software issue to a graph with different types of nodes and edges. Fourth, we construct a heterogeneous graph neural networks model with the support of fastText [6] for constructing initial node embedding to learn and predict the story points of new issues.Results. We did the comparison over three scenarios of estimation, including within project, cross-project within the repository, and cross-project cross repository with our baseline approaches. We achieve the average Mean Absolute Error (MAE) as 2.38, 2.61, and 2.63 for three scenarios. We outperform GPT2SP in 2/3 of the scenarios while outperforming Deep-SE in the most challenging scenario with significantly less amount of running time. We also compare our approaches with different homogeneous graph neural network models and the results show that the heterogeneous graph neural networks model outperforms the homogeneous models in story point estimation. For time performance, we achieve about 570 seconds as the time performance in both three processes: node embedding initialization, model construction, and story point estimation. HeterpSP's artifacts are available at [22].Conclusion. HeteroSP, a heterogeneous graph neural networks model for story point estimation, achieved good accuracy and running time.

show abstract

“…In recent years, increasing attention and efforts have been devoted into graph neural networks, which successfully extend deep neural networks to graph data. Graph neural networks are theoretically and empirically demonstrated to be very powerful in graph representation learning [15,26,43], and have achieved great success in various applications from different domains, such as natural language processing [6,22,53], computer vision [36,47] and recommendation [18,24,46,49,52,54,60]. There are mainly two groups of graph neural networks: the spectral-based methods and the spatial-based methods.…”

Section: Graph Neural Networkmentioning

confidence: 99%

An Adaptive Graph Pre-training Framework for Localized Collaborative Filtering

Wang¹,

Li²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNNs) have been widely applied in the recommendation tasks and have obtained very appealing performance.However, most GNN-based recommendation methods suffer from the problem of data sparsity in practice. Meanwhile, pre-training techniques have achieved great success in mitigating data sparsity in various domains such as natural language processing (NLP) and computer vision (CV). Thus, graph pre-training has the great potential to alleviate data sparsity in GNN-based recommendations.However, pre-training GNNs for recommendations face unique challenges. For example, user-item interaction graphs in different recommendation tasks have distinct sets of users and items, and they often present different properties. Therefore, the successful mechanisms commonly used in NLP and CV to transfer knowledge from pre-training tasks to downstream tasks such as sharing learned embeddings or feature extractors are not directly applicable to existing GNN-based recommendations models. To tackle these challenges, we delicately design an adaptive graph pre-training framework for localized collaborative filtering (ADAPT). It does not require transferring user/item embeddings, and is able to capture both the common knowledge across different graphs and the uniqueness for each graph. Extensive experimental results have demonstrated the effectiveness and superiority of ADAPT.

show abstract

Text Level Graph Neural Network for Text Classification

Cited by 32 publications

References 17 publications

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis

Heterogeneous Graph Neural Networks for Software Effort Estimation

An Adaptive Graph Pre-training Framework for Localized Collaborative Filtering

Contact Info

Product

Resources

About