Design and FPGA Implementation of an Adaptive video Subsampling Algorithm for Energy-Efficient Single Object Tracking

Iqbal, Odrika; Siddiqui, Saquib Ahmad; Martin, Joshua; Katoch, Sameeksha; Spanias, Andreas; Bliss, Daniel W.; Jayasuriya, Suren

doi:10.1109/icip40778.2020.9191146

Cited by 10 publications

(17 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the last 3 years approximately 30 students have been trained through this program. About 30% of the students produced publishable results [25][26][27][28][29][30][31][32][33] and two students participated in patent disclosures [34][35]. The recruitment demographics of the program were exceptional and details have been reported in NSF annual reports [36].…”

Section: Reu In Sensors Devices and ML Algorithmsmentioning

confidence: 99%

“…The recruitment demographics of the program were exceptional and details have been reported in NSF annual reports [36]. REU projects included sensors for cancer detection, audio processing, breathing sensors, ion channel sensors and signal processing, activity detection, object detection [33], Crowd Sourced Environmental Monitoring, and Monitoring Childhood Asthma.…”

Section: Reu In Sensors Devices and ML Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning Workforce Development Programs on Health and COVID-19 Research

Spanias

2020

2020 11th International Conference on Information, Intelligence, Systems and Applications (IISA

Self Cite

View full text Add to dashboard Cite

This paper accompanies the keynote speech at IISA-2020 and describes federally funded workforce development research grants and supplements in the area of sensors and machine learning. These programs operate under the auspices of the Sensor Signal and Information Processing (SenSIP) center which is also an Industry University Cooperative Research Center (I/UCRC) sponsored by the National Science Foundation (NSF) and I/UCRC industry members. The first program is an NSF REU site which has trained more than 30 students working on sensor hardware design and machine learning algorithm development. The second program is the NSF IRES site which is collaborative with the University of Cyprus and is focused on sensors and machine learning for energy systems. The most recent program funded by NSF is a Research Experiences for Teachers (RET) program that started in June 2020. This program embeds teachers and community college faculty in SenSIP machine learning projects. Another state funded program in which SenSIP is a partner is MedTech ventures. Our partner MedTech works on training medical technology students, entrepreneurs and engineers to create smart medical solutions for preventive healthcare. SenSIP also received NSF supplements to train students in using machine learning for COVID-19 detection.

show abstract

Section: Reu In Sensors Devices and ML Algorithmsmentioning

confidence: 99%

Section: Reu In Sensors Devices and ML Algorithmsmentioning

confidence: 99%

Machine Learning Workforce Development Programs on Health and COVID-19 Research

Spanias

2020

2020 11th International Conference on Information, Intelligence, Systems and Applications (IISA

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this work, we show how we can couple off-the-shelf object detectors with a Kalman filter to jointly perform predictive object tracking and adaptive subsampling. Our paper builds on initial work presented in [28], and we extend that work by introducing several new types of object detectors to the adaptive subsampling pipeline, and evaluating these algorithms on more comprehensive test datasets. In addition, we also identify a suitable candidate for hardware acceleration, and map the neural network-based approach (Efficient Convolution Operators for Tracking (ECO) plus Kalman filter) onto an FPGA.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Subsampling for ROI-based Visual Tracking: Algorithms and FPGA Implementation

Iqbal¹,

Muro²,

Katoch³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

There is tremendous scope for improving the energy efficiency of embedded vision systems by incorporating programmable region-of-interest (ROI) readout in the image sensor design. In this work, we study how ROI programmability can be leveraged for tracking applications by anticipating where the ROI will be located in future frames and switching pixels off outside of this region. We refer to this process of ROI prediction and corresponding sensor configuration as adaptive subsampling. Our adaptive subsampling algorithms comprise an object detector and an ROI predictor (Kalman filter) which operate in conjunction to optimize the energy efficiency of the vision pipeline with the end task being object tracking. To further facilitate the implementation of our adaptive algorithms in real life, we select a candidate algorithm and map it onto an FPGA. Leveraging Xilinx Vitis AI tools, we designed and accelerated a YOLO object detector-based adaptive subsampling algorithm. In order to further improve the algorithm post-deployment, we evaluated several competing baselines on the OTB100 and LaSOT datasets. We found that coupling the ECO tracker with the Kalman filter has a competitive AUC score of 0.4568 and 0.3471 on the OTB100 and LaSOT datasets respectively. Further, the power efficiency of this algorithm is on par with, and in a couple of instances superior to, the other baselines. The ECO-based algorithm incurs a power consumption of approximately 4 W averaged across both datasets while the YOLO-based approach requires power consumption of approximately 6 W (as per our power consumption model). In terms of accuracy-latency tradeoff, the ECO-based algorithm provides near-real-time performance (19.23 FPS) while managing to attain competitive tracking precision.

show abstract

“…Due to design flexibility through partial reconfiguration [1]- [5] and fast prototyping [6] [7], small-scale Field Programmable Gate Arrays (FPGAs) [8] are being used to deploy Convolutional Neural Networks (CNNs) on Resource-Constrained (RC) devices in Artificial Intelligence of Things (AIoT), especially for Edge Intelligence (EI) based applications [9]. However, it is very challenging to achieve ef-ficient deployment of CNNs on small-scale FPGAs.…”

Section: Introductionmentioning

confidence: 99%

“…More recently, Artificial Intelligence of Things (AIoT) also warrants RC devices to compute CNN. Due to design flexibility through partial reconfiguration [2]- [6] and fast prototyping [7] [8], small-scale Field Programmable Gate Arrays (FPGAs) [9] are being used to deploy CNNs on RC devices in AIoT, especially for EI based applications [1]. However, it is very challenging to achieve efficient deployment of CNNs on small-scale FPGAs.…”

Section: Introductionmentioning

confidence: 99%

FeSHI: Feature Map-Based Stealthy Hardware Intrinsic Attack

et al. 2021

View full text Add to dashboard Cite

Convolutional Neural Networks (CNN) have shown impressive performance in computer vision, natural language processing, and many other applications, but they exhibit high computations and substantial memory requirements. To address these limitations, especially in resource-constrained devices, the use of cloud computing for CNNs is becoming more popular. This comes with privacy and latency concerns that have motivated the designers to develop embedded hardware accelerators for CNNs. However, designing a specialized accelerator increases the time-to-market and cost of production. Therefore, to reduce the time-to-market and access to state-of-the-art techniques, CNN hardware mapping and deployment on embedded accelerators are often outsourced to untrusted third parties, which is going to be more prevalent in futuristic artificial intelligence of things (AIoT) systems. These AIoT systems anticipates horizontal collaboration among different resource constrained AIoT node devices, where CNN layers are partitioned and these devices collaboratively compute complex CNN tasks. This horizontal collaboration opens another attack surface to the CNN-based application, like inserting the hardware Trojans (HT) into the embedded accelerators designed for the CNN. Therefore, there is a dire need to explore this attack surface for designing the secure embedded hardware accelerators for CNNs. Towards this goal, in this paper, we exploited this attack surface to propose an HT-based attack called FeSHI. Since in horizontal collaboration of RC AIoT devices different sections of CNN architectures are outsourced to different untrusted third parties, the attacker may not know the input image, but it has access to the layer-by-layer output feature maps information for the assigned sections of the CNN architecture. This attack exploits the statistical distribution, i.e., Gaussian distribution, of the layer-by-layer feature maps of the CNN to design two triggers for stealthy HT with a very low probability of triggering. Also three different novel, stealthy and effective trigger designs are proposed. To illustrate the effectiveness of the proposed attack, we deployed the LeNet and LeNet-3D on PYNQ to classify the MNIST and CIFAR-10 datasets, respectively, and tested FeSHI. The experimental results show that FeSHI utilizes up to 2% extra LUTs, and the overall resource overhead is less than 1% compared to the original designs. It is also demonstrated on the PYNQ board that FeSHI triggers the attack vary randomly making it extremely difficult to detect.

show abstract

Design and FPGA Implementation of an Adaptive video Subsampling Algorithm for Energy-Efficient Single Object Tracking

Cited by 10 publications

References 16 publications

Machine Learning Workforce Development Programs on Health and COVID-19 Research

Machine Learning Workforce Development Programs on Health and COVID-19 Research

Adaptive Subsampling for ROI-based Visual Tracking: Algorithms and FPGA Implementation

FeSHI: Feature Map-Based Stealthy Hardware Intrinsic Attack

Contact Info

Product

Resources

About