Unsupervised vs. Transfer Learning for Multimodal One-Shot Matching of Speech and Images

Abstract

We consider the task of multimodal one-shot speech-image matching. An agent is shown a picture along with a spoken word describing the object in the picture, e.g. cookie, broccoli and ice-cream. After observing one paired speech-image example per class, it is shown a new set of unseen pictures, and asked to pick the "ice-cream". Previous work attempted to tackle this problem using transfer learning: supervised models are trained on labelled background data not containing any of the one-shot classes. Here we compare transfer learning to unsupervised models trained on unlabelled in-domain data. On a dataset of paired isolated spoken and visual digits, we specifically compare unsupervised autoencoder-like models to supervised classifier and Siamese neural networks. In both unimodal and multimodal few-shot matching experiments, we find that transfer learning outperforms unsupervised training. We also present experiments towards combining the two methodologies, but find that transfer learning still performs best (despite idealised experiments showing the benefits of unsupervised learning).

Direct Multimodal Few-Shot Learning of Speech and Images

“…The first row is a naive indirect baseline where matching is performed on the input features (dynamic time warping (DTW or Dynamic Time Warping) over MFCCs for speech and cosine distance over image pixels). The best overall score of 85.5% is achieved by the direct MTriplet, giving an absolute improvement of more than 25% over the best previous result [8]. This model is followed by the direct MCAE.…”

“…Previous work [7,8] used a two-step indirect approach: a spoken query is compared to the spoken examples in the given support set of speech-image pairs, and the corresponding image is then used to select the closest item in the unseen matching set. The task is therefore reduced to two unimodal comparisons, with the support set acting as a pivot between the modalities.…”

“…com/LeanneNortje/direct_multimodal_few-shot_learning. (iii) The support set can be used in two ways: (a) an indirect matching approach [7,8] which uses two unimodal comparisons across the support set, and (b) the direct matching approach proposed in this paper.…”

“…The task is therefore reduced to two unimodal comparisons, with the support set acting as a pivot between the modalities. To do the unimodal comparisons, learned speech and image representations are used: [8] compared transfer learning to unsupervised learning. In transfer learning, representations are used from models trained on labelled background data (not containing any classes seen at test time).…”

Attention-Based Keyword Localisation in Speech Using Visual Grounding

“…One form of weak supervision is using images paired with spoken captions [7][8][9][10][11][12][13][14][15][16]. Compared to using labelled data, this form of visual supervision is closer to the types of signals that infants would have access to while learning their first language [17][18][19][20][21][22], and to how one would teach new words to robots using spoken language [23,24]. It is also conceivable that this type of visual supervision could be easier to obtain when developing systems for low-resource languages [25], e.g.…”