HarDNet: A Low Memory Traffic Network

Abstract

State-of-the-art neural network architectures such as ResNet, MobileNet, and DenseNet have achieved outstanding accuracy over low MACs and small model size counterparts. However, these metrics might not be accurate for predicting the inference time. We suggest that memory traffic for accessing intermediate feature maps can be a factor dominating the inference latency, especially in such tasks as real-time object detection and semantic segmentation of high-resolution video. We propose a Harmonic Densely Connected Network to achieve high efficiency in terms of both low MACs and memory traffic. The new network achieves 35%, 36%, 30%, 32%, and 45% inference time reduction compared with FC-DenseNet-103, DenseNet-264, ResNet-50, ResNet-152, and SSD-VGG, respectively. We use tools including Nvidia profiler and ARM Scale-Sim to measure the memory traffic and verify that the inference latency is indeed proportional to the memory traffic consumption and the proposed network consumes low memory traffic. We conclude that one should take memory traffic into consideration when designing neural network architectures for high-resolution applications at the edge.

Lightweight Prediction and Boundary Attention-Based Semantic Segmentation for Road Scene Understanding

“…Real-time semantic segmentation networks [29]- [31], [34]- [38] have attempted to find a good trade-off between speed and accuracy; the main purpose of these networks is to reduce the number of network parameters and FLOPS while minimizing the loss of accuracy. Zhao et al proposed a compressed-PSPNet-based image cascade network (ICNet) [37] that can perform real-time segmentation by extracting the semantic information in low-resolution images and the details in high-resolution images.…”

“…Wu et al introduced CGNet [29] composed of context guided blocks that can effectively combine local features with the contextual information, and CGNet has shown high performance without using a backbone network. Chao et al suggested memory traffic as a dominating factor of the inference latency, and proposed FC-HarDNet [31] that shows high efficiency in terms of FLOPS and memory traffic.…”

“…Next, because the proposed PBRM can be integrated into any semantic segmentation network with an encoder-decoder structure, we compared the performance of the semantic segmentation networks with and without using PBRM. In particular, we chose three state-of-the-art real-time semantic segmentation networks, CGNet [29], LiteSeg-mobile [30], and FC-HarDNet70 [31], and measured the performance on the Cityscapes and CamVid test sets. The accuracy on the Cityscapes test set was obtained by submitting our results to the official evaluation server 1 .…”

“…In the experiments, we compared the proposed PBRM with other attention modules proposed in state-of-the-art semantic segmentation networks, i.e., DANet [25] and CCNet [26], in terms of the number of parameters, FLOPS, and segmentation accuracy. Besides, we applied the proposed refinement module to several state-of-the-art real-time semantic segmentation networks including context guided network (CGNet) [29], LiteSeg-mobile [30], and fully-convolutional harmonic densely connected network (FC-HarDNet) [31] to verify the effectiveness of PBRM. Experimental results on the Cityscapes [32] and CamVid [33] datasets demonstrated that the proposed PBRM can further improve the segmentation accuracy of conventional networks with a slight increase in computational cost.…”

Deep Learning-based Denoising of TEMPEST Images for Efficient Optical Character Recognition

“…2) FCHardNet: The second model is a CNN with an encoder-decoder structure that uses HardNet as backbone [12]. It is based on the work by P.Chao on efficient image segmentation [13].…”