Detector-Free Weakly Supervised Grounding by Separation

Arbelle, Assaf; Doveh, Sivan; Alfassy, Amit; Shtok, Joseph; Lev, Guy; Schwartz, Eli; Kuehne, Hilde; Levi, Hila; Sattigeri, Prasanna; Panda, Rameswar; Chen, Chun-Fu; Bronstein, Alex; Saenko, Kate; Ullman, Shimon; Giryes, Raja; Feris, Rogério; Karlinsky, Leonid

doi:10.1109/iccv48922.2021.00182

Cited by 16 publications

(2 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During training, a random horizontal flip is applied, with 0.5 probability. All models are trained on a double GeForce RTX 2080Ti Nvidia GPU, while all experiments of Arbelle et al [3] were conducted on four V100 GPUs.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs

Shaharabany¹,

Tewel²,

Wolf³

2022

Preprint

View full text Add to dashboard Cite

Given an input image, and nothing else, our method returns the bounding boxes of objects in the image and phrases that describe the objects. This is achieved within an open world paradigm, in which the objects in the input image may not have been encountered during the training of the localization mechanism. Moreover, training takes place in a weakly supervised setting, where no bounding boxes are provided. To achieve this, our method combines two pre-trained networks: the CLIP image-to-text matching score and the BLIP image captioning tool. Training takes place on COCO images and their captions and is based on CLIP. Then, during inference, BLIP is used to generate a hypothesis regarding various regions of the current image. Our work generalizes weakly supervised segmentation and phrase grounding and is shown empirically to outperform the state of the art in both domains. It also shows very convincing results in the novel task of weakly-supervised open-world purely visual phrase-grounding presented in our work. For example, on the datasets used for benchmarking phrasegrounding, our method results in a very modest degradation in comparison to methods that employ human captions as an additional input. Our code is available at https://github.com/talshaharabany/what-is-where-by-looking and a live demo can be found at https://replicate.com/talshaharabany/ what-is-where-by-looking.

show abstract

Section: Methodsmentioning

confidence: 99%

“…It cannot be compared directly with our work, since it uses additional supervision. Arbelle et al [3] proposed a self-supervision method scheme for generating a localization map.…”

Section: Related Workmentioning

confidence: 99%