Evaluation of automatic video captioning using direct assessment

Graham, Yvette; Awad, George; Smeaton, Alan F.

doi:10.1371/journal.pone.0202789

Cited by 20 publications

(16 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The remaining 69%, who did not meet this criterion, were omitted from computation of the of-ficial DA results above. Of those 31% included in the evaluation, a very high proportion, 97%, showed no significant difference in scores collected in repeated assessment of the same sentences; these high levels of agreement are consistent with what we have seen in DA used for Machine Translation (Graham et al, 2016) and Video Captioning evaluation (Graham et al, 2017).…”

Section: Mechanical Turk Resultssupporting

confidence: 83%

The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results

Mille¹,

Belz²,

Bohnet³

et al. 2018

Proceedings of the First Workshop on Multilingual Surface Realisation

Self Cite

View full text Add to dashboard Cite

We report results from the SR'18 Shared Task, a new multilingual surface realisation task organised as part of the ACL'18 Workshop on Multilingual Surface Realisation. As in its English-only predecessor task SR'11, the shared task comprised two tracks with different levels of complexity: (a) a shallow track where the inputs were full UD structures with word order information removed and tokens lemmatised; and (b) a deep track where additionally, functional words and morphological information were removed. The shallow track was offered in ten, and the deep track in three languages. Systems were evaluated (a) automatically, using a range of intrinsic metrics, and (b) by human judges in terms of readability and meaning similarity. This report presents the evaluation results, along with descriptions of the SR'18 tracks, data and evaluation methods. For full descriptions of the participating systems, please see the separate system reports elsewhere in this volume.

show abstract

Section: Mechanical Turk Resultssupporting

confidence: 83%

The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results

Mille¹,

Belz²,

Bohnet³

et al. 2018

Proceedings of the First Workshop on Multilingual Surface Realisation

Self Cite

View full text Add to dashboard Cite

show abstract

“…The remaining 69%, who did not meet this criterion, were omitted from computation of the of-ficial DA results above. Of those 31% included in the evaluation, a very high proportion, 97%, showed no significant difference in scores collected in repeated assessment of the same sentences; these high levels of agreement are consistent with what we have seen in DA used for Machine Translation and Video Captioning evaluation (Graham et al, 2017).…”

Section: Mechanical Turk Resultssupporting

confidence: 84%

Proceedings of the First Workshop on Multilingual Surface Realisation

Mille

Belz²,

Bohnet

et al. 2018

View full text Add to dashboard Cite

Natural Language Generation (NLG) is in the ascendant both as a stand-alone (data-to-text or text-totext) task and as part of downstream applications such as abstractive summarization, dialogue-based interaction, question answering, etc. Only in 2017, three "deep" NLG shared tasks that focused on language generation from abstract semantic representations have been organized, although for English only. Surface realization is also a burning issue, in particular, in view of the recent creation of multilingual treebanks annotated with Universal Dependencies (UD). The Multilingual Surface Realization Shared Task (SR '18), whose outcome is presented in these proceedings, targets surface realization from data released for the recent CoNLL shared task on multilingual parsing to UDs. After the First Surface Realization Shared Task in 2011, which focused on English, SR '18 is the second shared task on surface realization and the first to target multilingual input. 21 teams registered for SR '18, and eight of them submitted outputs of their systems.These proceedings include an overview of SR '18 and the description of the eight participating systems, which will be presented at the workshop. We are pleased that for the workshop we could also win Hadar Shemtov, Head of NLG, dialog and summarization groups at Google Research, as invited speaker. We trust that overall the workshop will be a forum for fruitful discussion, and that it will give an impetus to further advances and to further shared tasks in the field.

show abstract

“…While we attempt to find these better metrics, there is a large amount of research being done to improve captioning technology, and they use the existing metrics to evaluate their performance. Recognizing the problem, at TRECVID, we decided to include a manual evaluation, known as Direct Assessment (DA) (Graham et al, 2018 ) to selected submissions. The basic methodology is to present human assessors with a video and a single caption.…”

Section: Automatic and Manual Evaluationmentioning

confidence: 99%