Disentangled Graph Recurrent Network for Document Ranking

Abstract

BERT-based ranking models are emerging for its superior natural language understanding ability. All word relations and representations in the concatenation of query and document are modeled in the self-attention matrix as latent knowledge. However, some latent knowledge has none or negative effect on the relevance prediction between query and document. We model the observable and unobservable confounding factors in a causal graph and perform do-query to predict the relevance label given an intervention over this graph. For the observed factors, we block the back door path by an adaptive masking method through the transformer layer and refine word representations over this disentangled word graph through the refinement layer. For the unobserved factors, we resolve the do-operation query from the front door path by decomposing word representations into query related and unrelated parts through the decomposition layer. Pairwise ranking loss is mainly used for the ad hoc document ranking task, triangle distance loss is introduced to both the transformer and refinement layers for more discriminative representations, and mutual information constraints are put on the decomposition layer. Experimental results on public benchmark datasets TREC Robust04 and WebTrack2009-12 show that DGRe outperforms state-of-the-art baselines more than 2% especially for short queries.

Federated Learning over Coupled Graphs

“…SGC [9] adopts straightforward low-pass filters to simplify GCN, which smooths the neighbor features using normalized adjacency matrices. GNN models, especially GCN and its variants, have been successfully applied to recommendation systems [41], [42], [43], social networks mining [44], [45], natural language processing [46], [47], [48], and biochemistry [49]. The most related line of research to our work is the FL over Graphs.…”

Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking

“…BERT [4], ERNIE [42] and RoBERTa [26], have dominated many natural language processing tasks, and have also achieved remarkable success on passage re-ranking. For example, PLM based re-rankers [5,6,23,28] have achieved state-of-the-art performance, which takes the concatenation of query-passage pair as input, and applies multi-layer full-attention to model their semantic relevance. Their superiority can be attributed to the expressive transformer structure and the pretrain-then-finetune paradigm, which allow the model to learn useful implicit knowledge (i.e., semantic relevance in the latent space) from massive textual corpus [8].…”