What Can We Do to Improve Peer Review in NLP?

Rogers, Anna; Augenstein, Isabelle

doi:10.18653/v1/2020.findings-emnlp.112

Cited by 25 publications

(24 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our fast-paced field rewards the contribution of new methods and state-of-the-art results (Rogers and Augenstein, 2020), which often contrasts with controlled comparisons and training multiple models for variance estimation. In this paper, we showed that several methods for vision-andlanguage representation learning do not significantly differ when compared in a controlled setting.…”

Section: Discussionmentioning

confidence: 99%

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

Bugliarello

Cotterell

Okazaki

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Large-scale pretraining and task-specific fine- tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorized into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five vision and language BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.

show abstract

Section: Discussionmentioning

confidence: 99%

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

Bugliarello

Cotterell

Okazaki

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Most task-specific tracks (question answering, summarization, dialogue etc.) are supposed to receive both engineering and data submissions, but in that setting the interdisciplinary tension may lead to resource papers voted down simply for being resource papers (Rogers and Augenstein, 2020). Bawden (2019) cites an ACL 2019 reviewer who complained that "the paper is mostly a description of the corpus and its collection and contains little scientific contribution".…”

Section: Moving Forwardmentioning

confidence: 99%

Changing the World by Changing the Data

Rogers¹

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

NLP community is currently investing a lot more research and resources into development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. Algorithmic solutions have so far had limited success. An alternative that is being actively discussed is more careful design of datasets so as to deliver specific signals. This position paper maps out the arguments for and against data curation, and argues that fundamentally the point is moot: curation already is and will be happening, and it is changing the world. The question is only how much thought we want to invest into that process.

show abstract

“…Once papers are written and submitted for peer review, it is pertinent to evaluate them fairly and objectively. This process is far from straight-forward, as, among others, reviewers have certain biases, including against truly novel research (Rogers and Augenstein, 2020;Bhattacharya and Packalen, 2020). Research has thus focused on automatically generating peer reviews from paper content , as well as on studying how well review scores can be predicted from review texts (Kang et al, 2018;Plank and van Dalen, 2019).…”

Section: The Life Cycle Of Scientific Researchmentioning

confidence: 99%

Determining the Credibility of Science Communication

Augenstein¹

2021

Proceedings of the Second Workshop on Scholarly Document Processing

Self Cite

View full text Add to dashboard Cite

Most work on scholarly document processing assumes that the information processed is trustworthy and factually correct. However, this is not always the case. There are two core challenges, which should be addressed: 1) ensuring that scientific publications are credible -e.g. that claims are not made without supporting evidence, and that all relevant supporting evidence is provided; and 2) that scientific findings are not misrepresented, distorted or outright misreported when communicated by journalists or the general public. I will present some first steps towards addressing these problems and outline remaining challenges.

show abstract

What Can We Do to Improve Peer Review in NLP?

Cited by 25 publications

References 27 publications

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

Changing the World by Changing the Data

Determining the Credibility of Science Communication

Contact Info

Product

Resources

About