Active Ordinal Querying for Tuplewise Similarity Learning

Abstract

Many machine learning tasks such as clustering, classification, and dataset search benefit from embedding data points in a space where distances reflect notions of relative similarity as perceived by humans. A common way to construct such an embedding is to request triplet similarity queries to an oracle, comparing two objects with respect to a reference. This work generalizes triplet queries to tuple queries of arbitrary size that ask an oracle to rank multiple objects against a reference, and introduces an efficient and robust adaptive selection method called InfoTuple that uses a novel approach to mutual information maximization. We show that the performance of InfoTuple at various tuple sizes exceeds that of the state-of-the-art adaptive triplet selection method on synthetic tests and new human response datasets, and empirically demonstrate the significant gains in efficiency and query consistency achieved by querying larger tuples instead of triplets.

Active metric learning and classification using similarity queries

“…Ranking. In the second simulation, we compare the performance of actively selected nearest neighbor queries against ranking queries [10]. We observe that nearest neighbor queries perform competitively to ranking queries, as illustrated in Fig.…”

“…where D Qn refers to the set of distances between the reference r n and each of t c n ∈ T n and N n−1 Qn represents the assumed normal distribution on D Qn with the mean and variance determined by the estimate of distances after n − 1 queries. Due to this normal distribution assumption, the entropy in (10) and the expectation in (11) are straightforward calculations. The full procedure is shown in Alg.…”

“…We perform a set of experiments which utilize MDS to learn representations of the items. In particular, we use this opportunity to compare the performance of NN queries against a more complex ranking query [10]. When comparing against ranking queries, it is important to note that we expect both actively selected and randomly selected ranking queries to outperform a nearest neighbor query of the same size on a per-query basis, as there is a discrepancy in the amount of information each query contains.…”

“…For the following experiments, Info-Ranking-C means the active selection method in [10] was used to select ranking queries each with a set T i of size C.…”

“…To the best of our knowledge, we are the first to study this query type. Similar ideas have been explored before, such as using UI configurations to collect multiple triplets at once [8], enforcing a class-similarity based quadruplet loss (one anchor, one positive point, two negative points) [9], and soliciting ranking queries [10]. Of these approaches, [9] is the most similar, but NN queries are 1) not confined to a particular fixed length, and 2) not restricted to using class information.…”