Towards Visually Prompted Keyword Localisation for Zero-Resource Spoken Languages

Nortje, Leanne; Kamper, Herman

doi:10.1109/slt54892.2023.10023079

Cited by 5 publications

(6 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2 and we call it MATTNET (Multimodal ATTention NETwork). We adapt the multimodal localising attention model of [19] that consists of an audio and a vision branch. For the vision branch, we replace ResNet50 [23] with an adaption of AlexNet [24] to encode an image input x vision into a sequence of embeddings y vision .…”

Section: A Word-to-image Attention Mechanismmentioning

confidence: 99%

“…Originally [22], additional linear layers were used after the image embeddings, but removing these did not impact performance. For the audio branch, we use the same audio subnetwork as [19] that consists of an acoustic network f acoustic which extracts speech features from a spoken input x audio . However, [19] takes an entire spoken utterance as x audio , whereas we use a single isolated spoken word.…”

Section: A Word-to-image Attention Mechanismmentioning

confidence: 99%

“…For the audio branch, we use the same audio subnetwork as [19] that consists of an acoustic network f acoustic which extracts speech features from a spoken input x audio . However, [19] takes an entire spoken utterance as x audio , whereas we use a single isolated spoken word. We also add a few linear layers to the BiLSTM network f BiLSTM to encode the speech features into a single audio embedding y audio , similar to acoustic word embeddings [25]- [28].…”

Section: A Word-to-image Attention Mechanismmentioning

confidence: 99%

“…To get this word-to-image attention mechanism, we take the keyword localising attention mechanism of [19] which detects whether certain keywords occur in both spoken utterances and images. However, we aim to only detect whether a single isolated spoken word occurs somewhere within an image.…”

Section: A Word-to-image Attention Mechanismmentioning

confidence: 99%

“…This multimodal attention network (MATTNET), takes a single word embedding and calculates its correspondence to each pixel embedding to learn how the word is depicted within an image. This is similar to the vision attention part of the model from [19], where the goal was to localise visual keywords in speech (not in a few-shot setting).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Direct Multimodal Few-Shot Learning of Speech and Images

Nortje¹,

Kamper²

2021

Interspeech 2021

View full text Add to dashboard Cite

We propose direct multimodal few-shot models that learn a shared embedding space of spoken words and images from only a few paired examples. Imagine an agent is shown an image along with a spoken word describing the object in the picture, e.g. pen, book and eraser. After observing a few paired examples of each class, the model is asked to identify the "book" in a set of unseen pictures. Previous work used a two-step indirect approach relying on learned unimodal representations: speech-speech and image-image comparisons are performed across the support set of given speech-image pairs. We propose two direct models which instead learn a single multimodal space where inputs from different modalities are directly comparable: a multimodal triplet network (MTriplet or Multimodal Triplet Network) and a multimodal correspondence autoencoder (MCAE or Multimodal Correspondence Autoencoder). To train these direct models, we mine speech-image pairs: the support set is used to pair up unlabelled in-domain speech and images. In a speech-to-image digit matching task, direct models outperform indirect models, with the MTriplet achieving the best multimodal five-shot accuracy. We show that the improvements are due to the combination of unsupervised and transfer learning in the direct models, and the absence of two-step compounding errors.

show abstract

Section: A Word-to-image Attention Mechanismmentioning

confidence: 99%