ExpertosLF: dynamic late fusion of CBIR systems using online learning with relevance feedback

Abstract

One of the main challenges in CBIR systems is to choose discriminative and compact features, among dozens, to represent the images under comparison. Over the years, a great effort has been made to combine multiple features, mainly using early, late, and hierarchical fusion techniques. Unveiling the perfect combination of features is highly domain-specific and dependent on the type of image. Thus, the process of designing a CBIR system for new datasets or domains involves a huge experimentation overhead, leading to multiple fine-tuned CBIR systems. It would be desirable to dynamically find the best combination of CBIR systems without needing to go through such extensive experimentation and without requiring previous domain knowledge. In this paper, we propose ExpertosLF, a model-agnostic interpretable late fusion technique based on online learning with expert advice, which dynamically combines CBIR systems without knowing a priori which ones are the best for a given domain. At each query, ExpertosLF takes advantage of user’s feedback to determine each CBIR contribution in the ensemble for the following queries. ExpertosLF produces an interpretable ensemble that is independent of the dataset and domain. Moreover, ExpertosLF is designed to be modular, and scalable. Experiments on 13 benchmark datasets from the Biomedical, Real, and Sketch domains revealed that: (i) ExpertosLF surpasses the performance of state of the art late-fusion techniques; (ii) it successfully and quickly converges to the performance of the best CBIR sets across domains without any previous domain knowledge (in most cases, fewer than 25 queries need to receive human feedback).

Hybrid Bag-of-Visual-Words and FeatureWiz Selection for Content-Based Visual Information Retrieval

“…They used the Chi-square quadratic distance function to measure the similarity between the query image and the dataset images. Alarcao et al [ 27 ] proposed an Expert, dynamic Late Feature (ExpertosLF) fusion system for CBIR based on online learning via end-user relevance feedback. At each query, ExpertosLF determines the contribution of each feature collection in the ensemble for the subsequent queries based on the advantage of the user’s feedback.…”