Mixhead: Breaking the low-rank bottleneck in multi-head attention language models

Zhang, Zhong; Shao, Nian; Gao, Chongming; Miao, Rui; Yang, Qinli; Shao, Junming

doi:10.1016/j.knosys.2021.108075

Cited by 14 publications

(8 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The computational process is almost similar to self‐attention [31]. We introduce the Softmaxes [32] to replace the Softmax [28], which can predict the relation weight for any two positions in the feature maps and acquire more accurate probability values. Since the features are pooled and downsampled, the smaller scale bring a few amount of computation.…”

Section: Methodsmentioning

confidence: 99%

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

Yang,

Lei,

Sang

2023

IET Image Processing

View full text Add to dashboard Cite

Deep neural networks significantly enhance the accuracy of the stereo‐based disparity estimation. Some current methods suffer from inefficient use of the global context information, which will lead to the loss of structural details in ill‐posed areas. To this end, a novel stereo network GAMNet is designed, composed of three core components (GDA, MPF, DCA) for estimating the depth prediction in challenging real‐world environments. First, a lightweight attention module is presented, integrating the global semantic cues for every feature position across the channel and spatial dimensions. Next, the MPF module is constructed to fuse the diverse semantic and contextual information from different levels of the feature pyramid. Finally, cost volume is aggregated with a stacked encoder‐decoder composed of the DCA module and 3D convolutions, filtering the transmission of matching clues and capturing the rich global contexts. Substantial experiments conducted on KITTI 2012, KITTI 2015, SceneFlow, and Middlebury‐v3 datasets manifest that GAMNet surpasses preceding methods with contour‐preserving disparity predictions. In addition, the first 3D scene reconstruction linear evaluation strategy on spatial grasping points for the end‐to‐end stereo networks in an unsupervised mode is proposed, and it is deployed on the designed robot vision‐guided system. In application experiments, the method can produce densely high‐precision 3D reconstructions to implement the grasping task in complex real‐world scenes, and achieve excellent robust performance with competitive inference efficiency.

show abstract

Section: Methodsmentioning

confidence: 99%

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

Yang,

Lei,

Sang

2023

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…After the edge encoder and path encoder, we adopt two linear projection layers for the output of edge encoder and longest path encoder to transform the edge and path feature into high-dimension space. We also set a heading talking layer [32] to overcome low bottleneck [33].…”

Section: Graphnovo Architecturementioning

confidence: 99%

Mitigating the missing fragmentation problem in de novo peptide sequencing with a two stage graph-based deep learning model

Mao

Zhang

Xin

et al. 2023

Preprint

View full text Add to dashboard Cite

Novel protein discovery and immunopeptidomics depend on highly sensitive de novo peptide sequencing with tandem mass spectrometry. Despite significant improvement using deep learning models, the missing fragmentation problem remains an important hurdle that significantly degrades the performance of de novo peptide sequencing. In this paper, we reveal that in the process of peptide prediction, missing fragmentation results in generating incorrect amino acids within those regions and causes error accumulation thereafter. To tackle this problem, we propose GraphNovo, a two-stage de novo peptide sequencing algorithm based on a graph neural network. GraphNovo focuses on finding the optimal path in the first stage to guide the sequence prediction in the second stage. Our experiments demonstrate that GraphNovo mitigates the effects of missing fragmentation and outperforms the state-of-the-art de novo peptide sequencing algorithms.

show abstract

“…We remark, however, that the embedding space d will always be much smaller than the number of images or texts, which means that we are actually imposing a low-rank constraint on the joint probability distribution. In NLP, this effect has been referred to as the "softmax bottleneck" [43]. We now consider a set of factors Z = Z 1 × .…”

Section: Linear Compositionality In Vlmsmentioning

confidence: 99%

Linear Spaces of Meanings: the Compositional Language of VLMs

Trager¹,

Perera²,

Zancato³

et al. 2023

Preprint

View full text Add to dashboard Cite

We investigate compositional structures in vector data embeddings from pre-trained vision-language models (VLMs). Traditionally, compositionality has been associated with algebraic operations on embeddings of words from a pre-existing vocabulary. In contrast, we seek to approximate label representations from a text encoder as combinations of a smaller set of vectors in the embedding space. These vectors can be seen as "ideal words" which can be used to generate new concepts in an efficient way. We present a theoretical framework for understanding linear compositionality, drawing connections with mathematical representation theory and previous definitions of disentanglement. We provide theoretical and empirical evidence that ideal words provide good compositional approximations of composite concepts and can be more effective than token-based decompositions of the same concepts.

show abstract

Mixhead: Breaking the low-rank bottleneck in multi-head attention language models

Cited by 14 publications

References 19 publications

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

Mitigating the missing fragmentation problem in de novo peptide sequencing with a two stage graph-based deep learning model

Linear Spaces of Meanings: the Compositional Language of VLMs

Contact Info

Product

Resources

About