Raison d’être of the benchmark dataset: A Survey of Current Practices of Benchmark Dataset Sharing Platforms

Abstract

This paper critically examines the current practices of benchmark dataset sharing in NLP and suggests a better way to inform reusers of the benchmark dataset. As the dataset sharing platform plays a key role not only in distributing the dataset but also in informing the potential reusers about the dataset, we believe data sharing platforms should provide a comprehensive context of the datasets. We survey four benchmark dataset sharing platforms: HuggingFace, PaperswithCode, Tensorflow, and Pytorch to diagnose the current practices of how the dataset is shared -which metadata is shared and omitted. To be specific, drawing on the concept of data curation which considers the future reuse when the data is made public, we advance the direction that benchmark dataset sharing platforms should take into consideration. We identify that four benchmark platforms have different practices of using metadata and there is a lack of consensus on what social impact metadata is. We believe the problem of missing a discussion around social impact in the dataset sharing platforms has to do with the failed agreement on who should be in charge. We propose that the benchmark dataset should develop social impact metadata and data curator should take a role in managing the social impact metadata.

Exploring an Online Community of Blind Programmers by Using Topic Modeling and Network Analysis

“…We estimated the knowledge by topics of the thread that askers and answerers collaborated on to address RQ 1. We used the topic modeling algorithm which takes word embeddings and estimates distinctive topics in the vector space (Angelov, 2020), which was enabled through several deep learning natural language processing (NLP) frameworks (Park & Jeoung, 2022). As we used a thread as the unit of analysis and the topic vector will be given to threads, we can estimate which topic the users asked and answered.…”