Video Search with Sub-Image Keyword Transfer Using Existing Image Archives

Hezel, Nico; Schall, Konstantin; Jung, Klaus; Barthel, Kai Uwe

doi:10.1007/978-3-030-67835-7_49

Cited by 7 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HTW enables browsing of the whole video collection on keyframe or shot level by arranging the images on a hierarchical self-sorting map (SSM) [18]. Furthermore, the top-2000 results are presented in either a list, a hierarchical SSM or video summary consisting of five shots.…”

Section: Results Set Visualization and Browsingmentioning

confidence: 99%

“…VIREO utilized the decoded concept list of visual embedding [74]. HTW uses tagged image archives [18] to generate concepts, and Exquisitor uses pylucene to search the ResNeXt-101 visual concepts and their text descriptions [76] to provide positive examples to its relevance feedback process.…”

Section: Text Searchmentioning

confidence: 99%

“…VIREO uses visual embeddings of the dual-task model [74], VERGE the last pooling layer of a fine-tuned GoogleNet [45], HTW a CNN with DARAC-Pooling [61] and VISIONE Resnet101-GeM [50] and TERN [39]. For color or semantic sketches, vitrivr supports a plethora of features [51,56], VERGE clusters to twelve predefined colors using the Color Layout MPEG-7 descriptor, and HTW uses a handcrafted low-level feature [18]. VIREO [42] and VISIONE [1] also support sketch search, with VISIONE extracting dominant colors with pretrained color hash tables [5,72] and objects using pretrained neural networks [47,48,78].…”

Section: Image and Sketch Searchmentioning

confidence: 99%

See 2 more Smart Citations

Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

Heller

Gsteiger

Bailer

et al. 2022

Int J Multimed Info Retr

Self Cite

View full text Add to dashboard Cite

The Video Browser Showdown addresses difficult video search challenges through an annual interactive evaluation campaign attracting research teams focusing on interactive video retrieval. The campaign aims to provide insights into the performance of participating interactive video retrieval systems, tested by selected search tasks on large video collections. For the first time in its ten year history, the Video Browser Showdown 2021 was organized in a fully remote setting and hosted a record number of sixteen scoring systems. In this paper, we describe the competition setting, tasks and results and give an overview of state-of-the-art methods used by the competing systems. By looking at query result logs provided by ten systems, we analyze differences in retrieval model performances and browsing times before a correct submission. Through advances in data gathering methodology and tools, we provide a comprehensive analysis of ad-hoc video search tasks, discuss results, task design and methodological challenges. We highlight that almost all top performing systems utilize some sort of joint embedding for text-image retrieval and enable specification of temporal context in queries for known-item search. Whereas a combination of these techniques drive the currently top performing systems, we identify several future challenges for interactive video search engines and the Video Browser Showdown competition itself.

show abstract

Section: Results Set Visualization and Browsingmentioning

confidence: 99%

Section: Text Searchmentioning

confidence: 99%

Section: Image and Sketch Searchmentioning

confidence: 99%

See 1 more Smart Citation

Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

Heller

Gsteiger

Bailer

et al. 2022

Int J Multimed Info Retr

Self Cite

View full text Add to dashboard Cite

show abstract

“…RS is widely used in vector analytics, such as face recognition [62], near-duplicate detection [2], and various applications on general embeddings [58,65,69]. In some retrieval systems [14,56], ANNS is followed by RS with seed vertices and similarity thresholds [31] to obtain a similar cluster and offer more comprehensive results [39]. For example, users can choose a seed vertex to get all related results.…”

Section: Discussionmentioning

confidence: 99%

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment

Wang,

Xu,

et al. 2024

Proc. ACM Manag. Data

View full text Add to dashboard Cite

High-dimensional vector similarity search (HVSS) is gaining prominence as a powerful tool for various data science and AI applications. As vector data scales up, in-memory indexes pose a significant challenge due to the substantial increase in main memory requirements. A potential solution involves leveraging disk-based implementation, which stores and searches vector data on high-performance devices like NVMe SSDs. However, implementing HVSS for data segments proves to be intricate in vector databases where a single machine comprises multiple segments for system scalability. In this context, each segment operates with limited memory and disk space, necessitating a delicate balance between accuracy, efficiency, and space cost. Existing disk-based methods fall short as they do not holistically address all these requirements simultaneously. In this paper, we present Starling, an I/O-efficient disk-resident graph index framework that optimizes data layout and search strategy within the segment. It has two primary components: (1) a data layout incorporating an in-memory navigation graph and a reordered disk-based graph with enhanced locality, reducing the search path length and minimizing disk bandwidth wastage; and (2) a block search strategy designed to minimize costly disk I/O operations during vector query execution. Through extensive experiments, we validate the effectiveness, efficiency, and scalability of Starling. On a data segment with 2GB memory and 10GB disk capacity, Starling can accommodate up to 33 million vectors in 128 dimensions, offering HVSS with over 0.9 average precision and top-10 recall rate, and latency under 1 millisecond. The results showcase Starling's superior performance, exhibiting 43.9x higher throughput with 98% lower query latency compared to state-of-the-art methods while maintaining the same level of accuracy.

show abstract

“…vibro maintains its longstanding 2D map of visually arranged images, which has been improved in the 2023 version for faster and more accurate performance using Fast Linear Assignment Sorting (FLAS) [37] for grid layout arrangement and Dynamic Exploration Graph for internal graph representation [57]. These maps are mainly used to find similar images in the entire dataset, the current search result list, and the selected video.…”

Section: G Browsingmentioning

confidence: 99%

Evaluating Performance and Trends in Interactive Video Retrieval: Insights From the 12th VBS Competition

Vadicamo,

Arnold,

Bailer

et al. 2024

IEEE Access

View full text Add to dashboard Cite

This paper conducts a thorough examination of the 12th Video Browser Showdown (VBS) competition, which is a well-established international benchmarking campaign for interactive video search systems. The annual VBS competition has witnessed a steep rise in the popularity of multimodal embedding-based approaches in interactive video retrieval. The majority of the thirteen systems participating in VBS 2023 utilized a CLIP-based cross-modal search model, allowing the specification of free-form text queries to search visual content. This shared emphasis on joint embedding models contributed to balanced performance across various teams. However, the distinguishing factors of the top-performing teams included the adept combination of multiple models and search modes, along with the capabilities of interactive interfaces to facilitate and refine the search process. Our work provides an overview of the state-of-the-art approaches employed by the participating systems and conducts a thorough analysis of their search logs, which record user interactions and results of their queries for each task. Our comprehensive examination of the VBS competition offers assessments of the effectiveness of the retrieval models employed, the browsing efficiency, and user query patterns. Additionally, it provides valuable insights into the evolving landscape of interactive video retrieval and its future challenges.

show abstract

Video Search with Sub-Image Keyword Transfer Using Existing Image Archives

Cited by 7 publications

References 14 publications

Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment

Evaluating Performance and Trends in Interactive Video Retrieval: Insights From the 12th VBS Competition

Contact Info

Product

Resources

About