Augmenting Conversational Agents with Ambient Acoustic Contexts

Park, Chunjong; Min, Chulhong; Bhattacharya, Sourav; Kawsar, Fahim

doi:10.1145/3379503.3403535

Cited by 8 publications

(12 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For all datasets, we use the suggested train/test split for comparability purposes. For ambient sound classiication, we use the Ambient Acoustic Contexts dataset [33], in which sounds from ten distinct events are present. For the keyword spotting task, we use the second version of the Speech Commands dataset [41], where the objective is to detect when a particular keyword is spoken out of a set of twelve target classes.…”

Section: Datasets and Audio Pre-processingmentioning

confidence: 99%

“…Ambient Context [33] Event classiication 10 Speech Commands [41] Keyword spotting 12 VoxForge [29] Language identiication 6…”

Section: Datasetmentioning

confidence: 99%

“…• We exploit self-supervised models pre-trained on FSD-50K corpus [6] for signiicantly improving training convergence in federated settings. • We demonstrate through extensive evaluation that our technique is able to efectively learn generalizable audio models under a variety of federated settings and label availability on diverse public datasets, namely Speech Commands [41], Ambient Context [33] and VoxForge [29]. • We show that FedSTAR, with as few as 3% labeled data, on average can improve recognition rate by 13.28% across all datasets compared to the fully-supervised federated models.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Federated Self-training for Semi-supervised Audio Recognition

Tsouvalas

Saeed

Özçelebi

2022

ACM Trans. Embed. Comput. Syst.

View full text Add to dashboard Cite

Federated Learning is a distributed machine learning paradigm dealing with decentralized and personal datasets. Since data reside on devices like smartphones and virtual assistants, labeling is entrusted to the clients or labels are extracted in an automated way. Specifically, in the case of audio data, acquiring semantic annotations can be prohibitively expensive and time-consuming. As a result, an abundance of audio data remains unlabeled and unexploited on users’ devices. Most existing federated learning approaches focus on supervised learning without harnessing the unlabeled data. In this work, we study the problem of semi-supervised learning of audio models via self-training in conjunction with federated learning. We propose FedSTAR to exploit large-scale on-device unlabeled data to improve the generalization of audio recognition models. We further demonstrate that self-supervised pre-trained models can accelerate the training of on-device models, significantly improving convergence within fewer training rounds. We conduct experiments on diverse public audio classification datasets and investigate the performance of our models under varying percentages of labeled and unlabeled data. Notably, we show that with as little as 3% labeled data available, FedSTAR on average can improve the recognition rate by 13.28% compared to the fully-supervised federated model.

show abstract

Section: Datasets and Audio Pre-processingmentioning

confidence: 99%

“…Ambient Context [33] Event classiication 10 Speech Commands [41] Keyword spotting 12 VoxForge [29] Language identiication 6…”

Section: Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Federated Self-training for Semi-supervised Audio Recognition

Tsouvalas

Saeed

Özçelebi

2022

ACM Trans. Embed. Comput. Syst.

View full text Add to dashboard Cite

show abstract

“…Note that modelling all these context primitives, especially around the head, is not possible with other wearables. The acoustic channel of an earable enables us to understand environment ambience and audio events [19,23], thereby modelling a patient's proxemic, social context as well patient's interaction with physical objects [10]. Combining these primitives and their thoughtful synthesis is key in modelling patient activities and creating digital memories through encoding for future recall.…”

Section: Earable As Memory Aidmentioning

confidence: 99%

Designing Memory Aids for Dementia Patients using Earables

Franklin¹,

Lagnado²,

Min

et al. 2021

Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of The

Self Cite

View full text Add to dashboard Cite

Globally around 50 million people are currently living with dementia, and there are nearly 10 million new cases every year. The decline of memory and, with it, lack of self-confidence and continuous confusion have a devastating effect on people living with this disease. Dementia patients even struggle to accomplish mundane chores and require assistance for daily living and social connectedness. Over the past decade, we have seen remarkable growth in wearable technologies to manage our health and wellbeing and improve our awareness and social connectedness. However, we have to ask why wearables are not addressing this fundamental challenge of memory augmentation that threatens our society? Some limited existing work on cognitive wearables for dementia has focused on using images via camera-based life-logging technology. Instead, in this paper, we argue that earable -by virtue of its unique placement, rich sensing modalities, and acoustic feedback capabilities, uncovers new opportunities to augment human cognition to address this pressing need to assist dementia patients. To this end, we delve into fundamental principles of cognitive neuroscience to understand what constitutes memory disorder and its symptoms concerning errors in everyday activities. Building on this, we discuss the benefits of earables (in conjunction with smart objects) in modelling activity and intention of dementia patients and providing contextual memory cues. We put forward a guidance system to assist dementia patients with daily living and social connectedness. CCS CONCEPTS• Computer systems organization → Embedded systems.

show abstract

“…• We introduce adaptive confidence thresholding for generating pseudo-labels, which effectively reduces the number of inaccurate predicted annotations. • We demonstrate through extensive evaluation that our technique is able to effectively learn generalizable audio models under a variety of federated settings and label availability on diverse public datasets, namely Speech Commands [40], Ambient Context [32] and Vox-Forge [28]. • We exploit self-supervised models pre-trained on FSD-50K corpus [6] for significantly improving training convergence in federated settings.…”

Section: Introductionmentioning

confidence: 99%

Federated Self-Training for Semi-Supervised Audio Recognition

Tsouvalas¹,

Saeed²,

Özçelebi³

2021

Preprint

View full text Add to dashboard Cite

Federated Learning is a distributed machine learning paradigm dealing with decentralized and personal datasets. Since data reside on devices like smartphones and virtual assistants, labeling is entrusted to the clients or labels are extracted in an automated way. Specifically, in the case of audio data, acquiring semantic annotations can be prohibitively expensive and time-consuming. As a result, an abundance of audio data remains unlabeled and unexploited on users' devices. Most existing federated learning approaches focus on supervised learning without harnessing the unlabeled data. In this work, we study the problem of semi-supervised learning of audio models via self-training in conjunction with federated learning. We propose FedSTAR 1 to exploit large-scale on-device unlabeled data to improve the generalization of audio recognition models. We further demonstrate that self-supervised pre-trained models can accelerate the training of on-device models, significantly improving convergence to within fewer training rounds. We conduct experiments on diverse public audio classification datasets and investigate the performance of our models under varying percentages of labeled and unlabeled data. Notably, we show that with as little as 3% labeled data available, FedSTAR on average can improve the recognition rate by 13.28% compared to the fully-supervised federated model. CCS Concepts: • Computing methodologies → Semi-supervised learning settings; Neural networks; • Human-centered computing → Ubiquitous and mobile computing.

show abstract

Augmenting Conversational Agents with Ambient Acoustic Contexts

Cited by 8 publications

References 30 publications

Federated Self-training for Semi-supervised Audio Recognition

Federated Self-training for Semi-supervised Audio Recognition

Designing Memory Aids for Dementia Patients using Earables

Federated Self-Training for Semi-Supervised Audio Recognition

Contact Info

Product

Resources

About