EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications

Maaz, Muhammad; Shaker, Abdelrahman; Cholakkal, Hisham; Khan, Salman; Zamir, Syed Waqas; Anwer, Rao Muhammad; Khan, Fahad Shahbaz

doi:10.1007/978-3-031-25082-8_1

Cited by 125 publications

(45 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Surface normal calculation: project the 2D point pi,j onto the 3D surface, corresponding to the point Pi,j. The eight-neighbor point constraint method 19 is used to determine the plane around the pixel point pi,j. Then, the surface normals ng of the four corresponding planes are calculated in the 3D coordinate system, g=1, 2, 3, 4.…”

Section: Methodsmentioning

confidence: 99%

“…Figure 4 shows the location of DGCs in the network. The calculation process of absolute depth estimation is divided into four steps: 19 (1) Surface normal calculation: project the 2D point p i;j onto the 3D surface, corresponding to the point P i;j . The eight-neighbor point constraint method 19 is used to determine the plane around the pixel point p i;j .…”

Section: Dense Geometry Constraint Modulementioning

confidence: 99%

“…Brostow et al 3 solved the obstacle occlusion problem to a certain extent by replacing the original luminance reconstruction error with pixel-by-pixel re-projection error. With the introduction of vision transformer (ViT), 10 researchers have incorporated it into various vision tasks [17][18][19][20][21] with good results. However, the application of a transformer to depth estimation is still effective.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Lightweight monocular absolute depth estimation based on attention mechanism

Jin,

Tao,

Qian

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

To solve the problem of obtaining a higher accuracy at the expense of redundant models, we propose a network architecture. We utilize a lightweight network that retains the high-precision advantage of the transformer and effectively combines it with convolutional neural network. By greatly reducing the training parameters, this approach achieves high precision, making it well suited for deployment on edge devices. A detail highlight module (DHM) is added to effectively fuse information from multiple scales, making the depth of prediction more accurate and clearer. A dense geometric constraints module is introduced to recover accurate scale factors in autonomous driving without additional sensors. Experimental results demonstrate that our model improves the accuracy from 98.1% to 98.3% compared with Monodepth2, and the model parameters are reduced by about 80%.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Dense Geometry Constraint Modulementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Lightweight monocular absolute depth estimation based on attention mechanism

Jin,

Tao,

Qian

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

show abstract

“…Given the rapid advancement in AI, we acknowledge that the pipeline we select may not sustain peak performance. For example, multi-modal LLMs are equipped with vision capabilities [43,51,78], and dense video captioning models may improve rapidly by benefiting from large-scale pre-trained models [85]. Despite technological advances, our work provides enduring insights that transcend the specific models.…”

Section: Selecting Models and Constructing Pipelinesmentioning

confidence: 99%

TutoAI: a cross-domain framework for AI-assisted mixed-media tutorial creation on physical tasks

Chen,

Morariu,

Truong

et al. 2024

Proceedings of the CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Moreover, lightweight inference is hindered by the limited ability to compute global interactions in the spatial dimension due to input size constraints. Similarly, EdgeNeXt [21] combines depth-wise separable convolution and transposed attention mechanisms to introduce a split depth-wise transpose attention that enhances resource utilization. However, the structural design of this model is dependent on intricate submodules, such as Res2Net [22], ConvNeXt [23], and XCA [24].…”

Section: Introductionmentioning

confidence: 99%

MixMobileNet: A Mixed Mobile Network for Edge Vision Applications

Meng,

Wu,

Feng

et al. 2024

Electronics

View full text Add to dashboard Cite

Currently, vision transformers (ViTs) have rivaled comparable performance to convolutional neural networks (CNNs). However, the computational demands of the transformers’ self-attention mechanism pose challenges for their application on edge devices. Therefore, in this study, we propose a lightweight transformer-based network model called MixMobileNet. Similar to the ResNet block, this model only comprises a MixMobile block (MMb), which combines the efficient local inductive bias with the explicit modeling features of a transformer to achieve the fusion of the local–global feature interactions. For local, we propose the local-feature aggregation encoder (LFAE), which incorporates a PC2P (Partial-Conv→PWconv→PWconv) inverted bottleneck structure for residual connectivity. In particular, the kernel and channel scale are adaptive, reducing feature redundancy in adjacent layers and efficiently representing parameters. For global, we propose the global-feature aggregation encoder (GFAE), which employs a pooling strategy and computes the covariance matrix between channels instead of the spatial dimensions, changing the computational complexity from quadratic to linear, and this accelerates the inference of the model. We perform extensive image classification, object detection, and segmentation experiments to validate model performance. Our MixMobileNet-XXS/XS/S achieves 70.6%/75.1%/78.8% top-1 accuracy with 1.5 M/3.2 M/7.3 M parameters and 0.2 G/0.5 G/1.2 G FLOPs on ImageNet-1K, outperforming MobileViT-XXS/XS/S with an improvement of +1.6%↑/+0.4%↑/+0.4%↑ with −38.8%↓/−51.5%↓/−39.8%↓ reduction in FLOPs. In addition, the MixMobileNet-S assembly of SSDLite and DeepLabv3 achieves an accuracy of 28.5 mAP/79.5 mIoU at COCO2017/VOC2012 with lower computation, demonstrating the competitive performance of our lightweight model.

show abstract

EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications

Cited by 125 publications

References 29 publications

Lightweight monocular absolute depth estimation based on attention mechanism

Lightweight monocular absolute depth estimation based on attention mechanism

TutoAI: a cross-domain framework for AI-assisted mixed-media tutorial creation on physical tasks

MixMobileNet: A Mixed Mobile Network for Edge Vision Applications

Contact Info

Product

Resources

About