StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation

Maharana, Adyasha; Hannan, Darryl; Bansal, Mohit

doi:10.1007/978-3-031-19836-6_5

Cited by 30 publications

(27 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address challenges in human evaluation due to ad-hoc practices, prior studies offer shared human evaluation protocols, such as the one for unconditional image synthesis [39]. NLG community provides in-depth analysis of challenges in human evaluation on story generation [16]. To facilitate reliable model comparison based on human evaluation, a platform hosting human evaluation of multiple language generation tasks is proposed [18].…”

Section: Related Workmentioning

confidence: 99%

“…Human evaluation is typically done in a crowdsourcing platform, and there are many well-known practices. Yet, quality control is an open challenge [16]. Recently, a platform for benchmarking multiple NLG tasks was launched [18].…”

Section: Lack Of Quality Checkmentioning

confidence: 99%

“…For example, the evaluation that produces relative rankings is difficult to interpret as each sample's rating depends on other samples. For better interpretability, direct scoring is preferred in recent human evaluations for natural language generation [6,16,18].…”

Section: Our Design For Text-to-image Evaluationmentioning

confidence: 99%

“…As we are not fully aware of the content in generated images, we always use the maturity qualification. Although location qualification iii) is recommended in prior work in NLG [16,18], VPN can easily cheat this qualification and is a common practice among annotators [17]. Also, it should be noted that a substantial group of residents in the US do not use English as their primary language.…”

Section: Qualificationsmentioning

confidence: 99%

See 3 more Smart Citations

Optimal Correction Cost for Object Detection Evaluation

Otani

Togashi

Nakashima

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Mean Average Precision (mAP) is the primary evaluation measure for object detection. Although object detection has a broad range of applications, mAP evaluates detectors in terms of the performance of ranked instance retrieval. Such the assumption for the evaluation task does not suit some downstream tasks. To alleviate the gap between downstream tasks and the evaluation scenario, we propose Optimal Correction Cost (OC-cost), which assesses detection accuracy at image level. OC-cost computes the cost of correcting detections to ground truths as a measure of accuracy. The cost is obtained by solving an optimal transportation problem between the detections and the ground truths. Unlike mAP, OC-cost is designed to penalize false positive and false negative detections properly, and every image in a dataset is treated equally. Our experimental result validates that OC-cost has better agreement with human preference than a ranking-based measure, i.e., mAP for a single image. We also show that detectors' rankings by OC-cost are more consistent on different data splits than mAP. Our goal is not to replace mAP with OC-cost but provide an additional tool to evaluate detectors from another aspect. To help future researchers and developers choose a target measure, we provide a series of experiments to clarify how mAP and OC-cost differ.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Lack Of Quality Checkmentioning

confidence: 99%

Section: Our Design For Text-to-image Evaluationmentioning

confidence: 99%

Section: Qualificationsmentioning

confidence: 99%

See 2 more Smart Citations

Optimal Correction Cost for Object Detection Evaluation

Otani

Togashi

Nakashima

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Another pioneering work in the field of story synthesis is StoryGAN [24], a conditional GAN [29] trained to generate visual sequences corresponding to text descriptions of a story. Subsequently, StoryDALL-E [28] proposed the adaptation of pre-trained text-to-image transformers to generate visual sequences that continue a given text-based story, introducing a new task of story continuation. More recently, Pan et al [32] proposed history-aware Auto-Regressive Latent Diffusion Models that leverage diffusion models for story synthesis, utilizing a history-aware conditioning network to encode past caption-image pairs.…”

Section: Introductionmentioning

confidence: 99%