Semantic Sketch-Based Video Retrieval with Autocompletion

Tanase, Claudiu; Giangreco, Ivan; Rossetto, Luca; Schuldt, Heiko; Seddati, Omar; Dupont, Stéphane; Altiok, Ozan Can; Sezgin, Metin

doi:10.1145/2876456.2879473

Cited by 7 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…sketch-text generation [22], sketch animation [44], sketch-based user interface retrieval [23], etc.) More related to our work are DuetDraw [37] and IMOTION [46]. DuetDraw presents an interface where AI and humans can draw pictures interactively to conduct further sketch colorization, showing great user experience and promising application prospects in human-AI co-creation.…”

Section: Related Workmentioning

confidence: 99%

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

Liu

Deng

Song

et al. 2022

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

Sketching is an intuitive and simple way to depict sciences with various object form and appearance characteristics. In the past few years, widely available touchscreen devices have increasingly made sketch-based human-AI co-creation applications popular. One key issue of sketch-oriented interaction is to prepare input sketches efficiently by non-professionals because it is usually difficult and time-consuming to draw an ideal sketch with appropriate outlines and rich details, especially for novice users with no sketching skills, thus sketching brings great obstacles for sketch applications in daily life. On the other hand, hand-drawn sketches are scarce and hard to collect. Given the fact that there are several large-scale sketch datasets providing sketch data resources, but they usually have a limited number of objects and categories in sketch, and do not support users to collect new sketch materials according to their personal preferences. In addition, few sketch-related applications support the reuse of existing sketch elements. Thus, knowing how to extract sketches from existing drawings and effectively re-use them in interactive scene sketch composition will provide an elegant way for sketch-based image retrieval (SBIR) applications, which are widely used in various touch screen devices. In this study, we first conduct a study on current SBIR to better understand the main requirements and challenges in sketch-oriented applications. Then we develop the SketchMaker as an interactive sketch extraction and composition system to help users generate scene sketches via reusing object sketches in existing scene sketches with minimal manual intervention. Moreover, we demonstrate how SBIR improves from composited scene sketches to verify the performance of our interactive sketch processing system. We also include a sketch-based video localization task as an alternative application of our sketch composition scheme. Our pilot study shows that our system is effective and efficient, and provides a way to promote practical applications of sketches.

show abstract

Section: Related Workmentioning

confidence: 99%

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

Liu

Deng

Song

et al. 2022

ACM Trans. Interact. Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Either independently or in conjunction with these sketches, semantic information can be expressed by keywords or descriptions that are derived from manual or machine generated annotations [14,31]. There are some approaches that try to bridge the divide between the visual sketches and the textual annotations, by generating the semantic labels based on sketched input [6,30]. However, these methods differ mostly in the user-facing query formulation stage and do not offer any different or even richer query information.…”

Section: Related Workmentioning

confidence: 99%

Query by Semantic Sketch

Rossetto¹,

Gasser²,

Schuldt³

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Sketch-based query formulation is very common in image and video retrieval as these techniques often complement textual retrieval methods that are based on either manual or machine generated annotations. In this paper, we present a retrieval approach that allows to query visual media collections by sketching concept maps, thereby merging sketch-based retrieval with the search for semantic labels. Users can draw a spatial distribution of different concept labels, such as "sky", "sea" or "person" and then use these sketches to find images or video scenes that exhibit a similar distribution of these concepts. Hence, this approach does not only take the semantic concepts themselves into account, but also their semantic relations as well as their spatial context. The efficient vector representation enables efficient retrieval even in large multimedia collections. We have integrated the semantic sketch query mode into our retrieval engine vitrivr and demonstrated its effectiveness.

show abstract