Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement

Passos, Leandro Aparecido; Papa, João Paulo; Ser, Javier Del; Hussain, Amir; Adeel, Ahsan

doi:10.1016/j.inffus.2022.09.006

Cited by 24 publications

(3 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Battery life is also an important issue in the design of multi modal hearing aids as different components of these devices needs to be charged when used for an extended period of service. Though recent work from [17,18] has focused on developing energy efficient machine learning models, the issue of cost is very important for deploying AV HAT technologies.…”

Section: Cost and Battery Lifementioning

confidence: 99%

See 1 more Smart Citation

Socio-Technical Trust For Multi-Modal Hearing Assistive Technology

Williams

Azim

Piskopani

et al. 2023

2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)

View full text Add to dashboard Cite

The landscape of opportunity is rapidly changing for audio-visual (AV) hearing assistive technology. While hearing assistive devices, such as hearing aids, have traditionally been developed for populations of deaf and hard of hearing (DHH) communities, the ubiquitous use of in-ear technology and recent advances in edge computing are reformulating what drives research and development in this domain. With that comes new challenges to consider from the perspective of multiple different stakeholders. In this position paper, we elaborate on seven key socio-technical challenges that may impede the adoption of trustworthy multi-modal hearing assistive technologies. We also draw upon a recent survey being piloted in the UK to examine perceptions of trust for audio systems in the context of human rights. We strongly encourage the research community to consider trust as a factor in developing new AV assistive hearing technologies, as trust may ultimately drive adoption of this technology within broader society.

show abstract

Section: Cost and Battery Lifementioning

confidence: 99%

“…• e) safeguard users' rights and provide user control (art. [14][15][16][17][18][19][20][21] that allows them to request for the removal of personal data, have it transferred to another organisation or object to the processing of their personal data for certain reasons;…”

Section: Privacy and Security Concernsmentioning

confidence: 99%

Socio-Technical Trust For Multi-Modal Hearing Assistive Technology

Williams

Azim

Piskopani

et al. 2023

2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)

View full text Add to dashboard Cite

show abstract

“…Traditional SR systems rely on audio signals captured by microphones, which can be affected by background noise, distance, and some environmental factors [2], [3]. SR using audio-visual signals, which involves combining information from both audio and visual cues to identify or verify the speaker's identity, has gained increasing attention in recent years [4], [5]. While SR using audio-visual signals has the potential to improve the accuracy and reliability of SR systems [6]- [8], it is important to consider the limitations of this technology [9].…”

Section: Introductionmentioning

confidence: 99%

Privacy-Preserving Speaker Recognition Using Radars for Context Estimation in Future Multi-Modal Hearing Assistive Technologies

Farooq,

Ge,

Qayyum

et al. 2023

2023 IEEE International Radar Conference (RADAR)

Self Cite

View full text Add to dashboard Cite

Speaker recognition (SR) from speech can help determine the environmental context in multi-talker conversational scenarios to enable the design of context-aware multi-modal hearing assistive technology. In this paper, we argue that the use of wireless sensors such as radars can offer many benefits over conventional audio and visual sensors, such as not being afflicted by privacy and environmental issues, e.g., improper lighting, environmental noise, and potential security concerns of audio and video channels. Radar is relatively less explored and has many advantages over other contactless approaches, such as being more compact compared to RFID and having a better range and resolution than ultrasound and microwave sensors. To this end, we propose the use of ultrawideband radar coupled with a deep learning model for SR from silent speech to enable the design of future context-aware multimodal hearing assistive technology. We collected a dataset from five individuals with origins in Europe, Asia, and the United Kingdom. We obtained an average performance of approximately 82% in recognising an unknown person from a set of known people. This demonstrates that the radar has good potential to be used for privacy-preserving SR in multi-talker environments where audio-visual and other contactless techniques have limited capabilities.

show abstract

ARIF: An Adaptive Attention-Based Cross-Modal Representation Integration Framework

Liu,

Luo,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement

Cited by 24 publications

References 33 publications

Socio-Technical Trust For Multi-Modal Hearing Assistive Technology

Socio-Technical Trust For Multi-Modal Hearing Assistive Technology

Privacy-Preserving Speaker Recognition Using Radars for Context Estimation in Future Multi-Modal Hearing Assistive Technologies

ARIF: An Adaptive Attention-Based Cross-Modal Representation Integration Framework

Contact Info

Product

Resources

About