SG-Net: Syntax-Guided Machine Reading Comprehension

Abstract

For machine reading comprehension, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy passages and getting ride of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. To verify its effectiveness, the proposed SG-Net is applied to typical pre-trained language model BERT which is right based on a Transformer encoder. Extensive experiments on popular benchmarks including SQuAD 2.0 and RACE show that the proposed SG-Net design helps achieve substantial performance improvement over strong baselines.

Document-level Relation Extraction with Dual-tier Heterogeneous Graph

“…It directly proves the necessity of incorporating structural information and reasoning mechanism in document-level RE, and many kinds of basic encoder could be well integrated into our model due to the loosely-coupled architecture. We consider the performance gain on BERT-DRE is mainly because that DHG makes full use of the semantic information of BERT, and the graph structure complements the weakness of BERT in capturing long-range syntactic structure, which is consistent with the conclusion of some recent studies (Clark et al, 2019;Zhang et al, 2020b). Meanwhile, another phenomenon is that the DHG-based models outperform all baseline models in both intra-and inter-sentence scenarios, especially the inter-sentence setting, which demonstrates that the majority of DHG mainly comes from inter-sentence relational facts.…”

Contextual and Semantic Fusion Network for Multiple-Choice Reading Comprehension

“…From a practical point of view, finetuning a pre-trained model is resource-saving. Besides, a variety of studies show that fine-tuning a pre-trained model has a great impact on MCRC performance [8]- [10].…”

Empowering Real-Time Traffic Reporting Systems With NLP-Processed Social Media Data

“…To do so, we implement the QA model [16]. In fact, recently published QA models including BERT and XLNet [17] are shown to achieve human-like performance when tested on the Stanford Question Answering data set [18]. Once the detailed traffic information is obtained, we implement an automated system that converts collected posts from social media into real-time traffic information that are used to update navigation maps and warn drivers about any traffic events.…”