Environmental Sound Classification with Tiny Transformers in Noisy Edge Environments

Wyatt, Steven B.; Elliott, David; Aravamudan, Akshay; Otero, Carlos E.; Otero, Luis Daniel; Anagnostopoulos, Georgios C.; Smith, Anthony O.; Peter, Adrian M.; Jones, Wesley B.; Leung, Steven; Lam, Eric P.

doi:10.1109/wf-iot51360.2021.9596007

Cited by 16 publications

(13 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, both studies required feature extraction for inference and achieved classification accuracies of less than 80%. Wyatt et al [34] and Elliott et al [9] developed transformer-based models applicable to edge devices and tested them on the Raspberry Pi Zero and Samsung Galaxy S9 devices, respectively. These on-device ESC studies with lightweight architectures demonstrated the potential for ESC on edge devices.…”

Section: Related Workmentioning

confidence: 99%

Urban Noise Analysis and Emergency Detection System using Lightweight End-to-End Convolutional Neural Network

Park,

Yoo,

Lee

et al. 2023

INT J COMPUT COMMUN, Int. J. Comput. Commun. Control

View full text Add to dashboard Cite

In recent years, the application of deep learning to environmental sound classification (ESC) has received considerable attention owing to its powerful ability to recognize the context of urban sounds. In general, deep learning models with high accuracy require substantial computing and memory resources. Consequently, to apply complex deep learning models to ESC in the real world, model inference has been performed on cloud servers with powerful computing resources. However, heavy network traffic and security issues occur when inferences are performed on a cloud server. In addition, deploying a deep learning model trained on a single public ESC dataset may not be sufficient for classifying various classes of urban noise and emergency-related sounds. To address these problems, we propose an on-device, real-time urban sound monitoring system that can classify various urban sounds at low system construction costs. The proposed system consisted of an edge artificial intelligence (AI) node and a FIWARE-based server. To enable the real-time inference on a resource-constrained edge AI node, we developed a lightweight convolutional neural network (CNN) by adjusting the input and model configurations to achieve high accuracy with a low number of parameters. The model achieved 94.9% classification accuracy using only 331 K parameters on an integrated dataset that included 17 classes of urban noises and emergencies. Furthermore, a prototype of the proposed system was developed and evaluated to verify its feasibility. The prototype system was built at a cost of less than USD 50 and could perform the entire monitoring process every 3 s. We also visualized the sound monitoring data using Grafana on a FIWARE-based server.

show abstract

Section: Related Workmentioning

confidence: 99%

Urban Noise Analysis and Emergency Detection System using Lightweight End-to-End Convolutional Neural Network

Park,

Yoo,

Lee

et al. 2023

INT J COMPUT COMMUN, Int. J. Comput. Commun. Control

View full text Add to dashboard Cite

show abstract

“…Regarding Pure Transformers, Elliott et al [19], Wyatt et al [20], and Devlin et al [25] explored the advantages of Bidirectional-Encoder-Representations-from-Transformers (BERT)based models, based on the work of Vaswani et al [26], having as the input a given token summed with the position embeddings, in order to address the sound classification problem at the edge.…”

Section: Transformers For Audio Classificationmentioning

confidence: 99%

“…More recently, attention mechanisms have been incorporated to focus on semantically important parts of the sound under study [13][14][15][16][17]. Lately, solutions based on attention models [11,18], particularly on Transformers [18][19][20][21][22], are being explored.…”

Section: Introductionmentioning

confidence: 99%

Transformers for Urban Sound Classification—A Comprehensive Performance Evaluation

Nogueira

Oliveira

Machado

et al. 2022

Sensors

View full text Add to dashboard Cite

Many relevant sound events occur in urban scenarios, and robust classification models are required to identify abnormal and relevant events correctly. These models need to identify such events within valuable time, being effective and prompt. It is also essential to determine for how much time these events prevail. This article presents an extensive analysis developed to identify the best-performing model to successfully classify a broad set of sound events occurring in urban scenarios. Analysis and modelling of Transformer models were performed using available public datasets with different sets of sound classes. The Transformer models’ performance was compared to the one achieved by the baseline model and end-to-end convolutional models. Furthermore, the benefits of using pre-training from image and sound domains and data augmentation techniques were identified. Additionally, complementary methods that have been used to improve the models’ performance and good practices to obtain robust sound classification models were investigated. After an extensive evaluation, it was found that the most promising results were obtained by employing a Transformer model using a novel Adam optimizer with weight decay and transfer learning from the audio domain by reusing the weights from AudioSet, which led to an accuracy score of 89.8% for the UrbanSound8K dataset, 95.8% for the ESC-50 dataset, and 99% for the ESC-10 dataset, respectively.

show abstract

“…Cantarini et al [ 71 ] applied the harmonic percussive source separation technique to classify emergency siren sounds from road noise sounds. Wyatt et al [ 72 ] deployed a BERT-based environmental sound classification model on an RPi Zero to identify six different everyday sounds (Knock, Laugh, Keyboard Typing, Cough, Keys Jangling and Snap). All three studies mentioned above described standalone audio capture and recognition systems for varied environmental settings and did not describe a multimodal data acquisition system.…”

Section: Related Workmentioning

confidence: 99%

An Edge Computing and Ambient Data Capture System for Clinical and Home Environments

Suresha

Hegde

Jiang

et al. 2022

Sensors

View full text Add to dashboard Cite

The non-contact patient monitoring paradigm moves patient care into their homes and enables long-term patient studies. The challenge, however, is to make the system non-intrusive, privacy-preserving, and low-cost. To this end, we describe an open-source edge computing and ambient data capture system, developed using low-cost and readily available hardware. We describe five applications of our ambient data capture system. Namely: (1) Estimating occupancy and human activity phenotyping; (2) Medical equipment alarm classification; (3) Geolocation of humans in a built environment; (4) Ambient light logging; and (5) Ambient temperature and humidity logging. We obtained an accuracy of 94% for estimating occupancy from video. We stress-tested the alarm note classification in the absence and presence of speech and obtained micro averaged F1 scores of 0.98 and 0.93, respectively. The geolocation tracking provided a room-level accuracy of 98.7%. The root mean square error in the temperature sensor validation task was 0.3°C and for the humidity sensor, it was 1% Relative Humidity. The low-cost edge computing system presented here demonstrated the ability to capture and analyze a wide range of activities in a privacy-preserving manner in clinical and home environments and is able to provide key insights into the healthcare practices and patient behaviors.

show abstract

Environmental Sound Classification with Tiny Transformers in Noisy Edge Environments

Cited by 16 publications

References 18 publications

Urban Noise Analysis and Emergency Detection System using Lightweight End-to-End Convolutional Neural Network

Urban Noise Analysis and Emergency Detection System using Lightweight End-to-End Convolutional Neural Network

Transformers for Urban Sound Classification—A Comprehensive Performance Evaluation

An Edge Computing and Ambient Data Capture System for Clinical and Home Environments

Contact Info

Product

Resources

About