Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning

Tu, Yunbin; Li, Liang; Su, Li; Du, Junping; Lü, Ke; Huang, Qingming

doi:10.1109/tip.2023.3268004

Cited by 16 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While all methods suffer significant performance drops when evaluated on a dataset from a different domain, VIXEN-Q shows a better ability to generalize to new data by scoring the highest on the PSBattles dataset. After finetuning the model on Image Editing Request, VIXEN-C outperforms previous methods on most metrics, except B@4 of VARD (Tu et al 2023a).…”

Section: Resultsmentioning

confidence: 96%

“…DUDA (Park, Darrell, and Rohrbach 2019) instead computes image difference at CNN semantic level, improving the robustness against slight global changes. In M-VAM (Shi et al 2020) and VACC (Kim et al 2021), a view-point encoder is proposed to mitigate potential view-point difference and VARD (Tu et al 2023a) proposes a viewpoint invariant representation network to explicitly capture the change. Meanwhile, (Sun et al 2022) uses bidirectional encoding to improve change localization and NCT (Tu et al 2023b) aggregates neighboring features with a transformer.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

VIXEN: Visual Text Comparison Network for Image Difference Captioning

Black,

Shi,

Fan

et al. 2024

AAAI

View full text Add to dashboard Cite

We present VIXEN - a technique that succinctly summarizes in text the visual differences between a pair of images in order to highlight any content manipulation present. Our proposed network linearly maps image features in a pairwise manner, constructing a soft prompt for a pretrained large language model. We address the challenge of low volume of training data and lack of manipulation variety in existing image difference captioning (IDC) datasets by training on synthetically manipulated images from the recent InstructPix2Pix dataset generated via prompt-to-prompt editing framework. We augment this dataset with change summaries produced via GPT-3. We show that VIXEN produces state-of-the-art, comprehensible difference captions for diverse image contents and edit types, offering a potential mitigation against misinformation disseminated via manipulated image content. Code and data are available at http://github.com/alexblck/vixen

show abstract

Section: Resultsmentioning

confidence: 96%