NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Dhole, Kaustubh; Gangal, Varun; Gehrmann, Sebastian; Gupta, Aadesh; Li, Zhenhao; Mahamood, Saad; Abinaya, Mahendiran,; Mille, Simon; Shrivastava, Ashish; Tan, Samson; Wu, Tongshuang; Sohl-Dickstein, Jascha; Choi, Jinho D.; Hovy, Eduard; Dušek, Ondřej; Ruder, Sebastian; Anand, Sajant; Aneja, Nagender; Banjade, Rabin; Barthe, Lisa; Behnke, Hanna; Berlot-Attwell, Ian; Boyle, Connor; Brun, Catherine; Cabezudo, Marco Antonio Sobrevilla; Cahyawijaya, Samuel; Chapuis, Emile; Che, Wanxiang; Choudhary, Mukund; Clauss, Christian; Colombo, Pierre; Cornell, Filip; Dagan, Gautier; Das, Mayukh; Tanay, Dixit,; Dopierre, Thomas; Dray, Paul-Alexis; Dubey, Suchitra; Tatiana, Ekeinhor,; Giovanni, Marco Di; Goyal, Tanya; Gupta, Rajiv; Hamla, Louanes; Han, Sang Wook; Harel-Canada, Fabrice; Honoré, Antoine; Jindal, Ishan; Joniak, Przemyslaw K.; Kleyko, Denis; Kovatchev, Venelin; Krishna, Kalpesh; Kumar, Ashok; Langer, Stefan; Ryan, Lee, Seungjae; Levinson, Corey James; Liang, Hualou; Liang, Kaizhao; Liu, Zhexiong; Lukyanenko, Andrey; Marivate, Vukosi; Melo, Gerard de; Méoni, Simon; Meyer, Maxime; Mir, Afnan; Moosavi, Nafise Sadat; Muennighoff, Niklas; Hon, Mun, Timothy Sum; Murray, Kenton; Namysł, Marcin; Obedkova, Maria; Oli, Priti; Pasricha, Nivranshu; Pfister, Jan; Plant, Richard; Prabhu, Vinay; Pais, V.; Qin, Libo; Raji, Shahab; Kumar, Rajpoot, Pawan; Raunak, Vikas; Rinberg, Roy; Roberts, Nicolas; Rodríguez, Juan Diego; Roux, Claude; Vasconcellos, P. H. S.; Sai, Ananya B.; Schmidt, Robin; Scialom, Thomas; Sefara, Tshephisho Joseph; Shamsi, Saqib; Shen, Xudong; Shi, Haoyue; Shi, Yiwen; Shvets, Anna; Siegel, Nick; Sileo, Damien; Simon, Jamie; Singh, Chandan Deep; Sitelew, Roman; Soni, Praveen; Sorensen, Taylor; Soto, William; Srivastava, Aman; Srivatsa, KV Aditya; Sun, Tony; Varma, T, Mukund; A, Tabassum,; Tan, Fiona Anting; Teehan, Ryan; Tiwari, Mo; Tolkiehn, Marie; Athena, Wang,; Wang, Zijian; Wang, Gloria; Wang, Zijie; Wei, Fuxuan; Wilie, Bryan; Winata, Genta Indra; Guo, Shirong; Wydmański, Witold; Xie, Tianbao; Yaseen, Usama; Yee, Michael A.; Zhang, Jing; Zhang, Yue

doi:10.48550/arxiv.2112.02721

Cited by 5 publications

(7 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We manually inspected a sample of the publications with the most URIs to one of the four GHPs and found these publications tend to detail a software product or provide an overview of a topic, such as survey paper. The top three publications containing the most URIs to a GHP include 153 [4], 160 [1], and 896 [26] URIs to GitHub. Dhole et al [4] developed a software product and included URIs to the implementation of the features listed in the publication.…”

Section: Resultsmentioning

confidence: 99%

“…The top three publications containing the most URIs to a GHP include 153 [4], 160 [1], and 896 [26] URIs to GitHub. Dhole et al [4] developed a software product and included URIs to the implementation of the features listed in the publication. Agol et al [1] created an open-source package and linked to the implementation of the algorithms and processes described in the publication.…”

Section: Resultsmentioning

confidence: 99%

“…A study by Fäber [9] found that GitHub, 4 a Git Hosting Platform (GHP), was the most popular repository hosting platform. A GHP is a type of repository hosting platform made specifically for Git VCS.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Rise of GitHub in Scholarly Publications

Escamilla

Klein

Cooper

et al. 2022

Linking Theory and Practice of Digital Libraries

View full text Add to dashboard Cite

The definition of scholarly content has expanded to include the data and source code that contribute to a publication. While major archiving efforts to preserve conventional scholarly content, typically in PDFs (e.g., LOCKSS, CLOCKSS, Portico), are underway, no analogous effort has yet emerged to preserve the data and code referenced in those PDFs, particularly the scholarly code hosted online on Git Hosting Platforms (GHPs). Similarly, the Software Heritage Foundation is working to archive public source code, but there is value in archiving the issue threads, pull requests, and wikis that provide important context to the code while maintaining their original URLs. In current implementations, source code and its ephemera are not preserved, which presents a problem for scholarly projects where reproducibility matters. To understand and quantify the scope of this issue, we analyzed the use of GHP URIs in the arXiv and PMC corpora from January 2007 to December 2021. In total, there were 253,590 URIs to GitHub, SourceForge, Bitbucket, and GitLab repositories across the 2.66 million publications in the corpora. We found that GitHub, GitLab, SourceForge, and Bitbucket were collectively linked to 160 times in 2007 and 76,746 times in 2021. In 2021, one out of five publications in the arXiv corpus included a URI to GitHub. The complexity of GHPs like GitHub is not amenable to conventional Web archiving techniques. Therefore, the growing use of GHPs in scholarly publications points to an urgent and growing need for dedicated efforts to archive their holdings in order to preserve research code and its scholarly ephemera.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

The Rise of GitHub in Scholarly Publications

Escamilla

Klein

Cooper

et al. 2022

Linking Theory and Practice of Digital Libraries

View full text Add to dashboard Cite

show abstract

“…AI-art models might be exclusively seen as potential attackers on the most talented segments of the artistic society, but they will doubtlessly open up a level playing field for those who considered art out of reach. Besides, such democratization would also be reflected in crowd-sourced efforts (Bigham, Kulkarni, and Lasecki 2017;Kittur et al 2013Kittur et al , 2019Dhole et al 2021;Srivastava et al 2022) which would seek contributions to aid in developing large creative models fairly.…”

Section: Art Democratizationmentioning

confidence: 99%

Is AI Art Another Industrial Revolution in the Making?

Newton¹,

Dhole²

2023

Preprint

View full text Add to dashboard Cite

A major shift from skilled to unskilled workers was one of the many changes caused by the Industrial Revolution, when the switch to machines contributed to decline in the social and economic status of artisans, whose skills were dismembered into discrete actions by factory-line workers. We consider what may be an analogous computing technology: the recent introduction of AI-generated art software. AI art generators such as Dall-E and Midjourney can create fully rendered images based solely on a user's prompt, just at the click of a button. Some artists fear if the cheaper price and conveyorbelt speed that comes with AI-produced images is seen as an improvement to the current system, it may permanently change the way society values/views art and artists. In this article, we consider the implications that AI art generation introduces through a post-industrial revolution historical lens. We then reflect on the analogous issues that appear to arise as a result of the AI art revolution, and we conclude that the problems raised mirror those of industrialization, giving a vital glimpse into what may lie ahead.

show abstract

“…The multi-tasks evaluation setting. If single-tasks problems are quite common, to best understand weakness and model in real-world scenario, the community is heading towards more complex evaluations involving fine-grained evaluation (Liu et al, 2021) across several metrics (or criteria (Gardent et al, 2017;Yuan et al, 2019)) and several tasks (Wang et al, 2018;Zheng et al, 2021;Gehrmann et al, 2021;McMillan-Major et al, 2021;Dhole et al, 2021;Tay et al, 2020a). This is due to the increasing performance of deep neural networks, which are nowadays designed to generalize in a great variety of situations and to solve complex tasks (Silver et al, 2016).…”

Section: Work In Progressmentioning

confidence: 99%

What are the best systems? New perspectives on NLP Benchmarking

Colombo¹,

Noiry²,

Irurozki³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are instrumental in (i) assessing the progress of new methods along different axes and (ii) selecting the best systems for practical use. This is particularly the case for NLP with the development of large pre-trained models (e.g. GPT, BERT) that are expected to generalize well on a variety of tasks. While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures. However, this procedure can be problematic when the metrics are on a different scale, which may lead to spurious conclusions. This paper proposes a new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded. We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores (e.g. GLUE, EXTREM, SE-VAL, TAC, FLICKR). In particular, we show that our method yields different conclusions on stateof-the-art systems than the mean-aggregation procedure while being both more reliable and robust.

show abstract

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Cited by 5 publications

References 51 publications

The Rise of GitHub in Scholarly Publications

The Rise of GitHub in Scholarly Publications

Is AI Art Another Industrial Revolution in the Making?

What are the best systems? New perspectives on NLP Benchmarking

Contact Info

Product

Resources

About