Assessing Annotation Consistency in the Wild

Abstract

The process of human annotation of sensor data is at the base of research areas such as participatory sensing and mobile crowdsensing. While much research has been devoted to assessing the quality of sensor data, the same cannot be said about annotations, which are fundamental to obtain a clear understanding of users experience. We present an evaluation of an interdisciplinary annotation methodology allowing users to continuously annotate their everyday life. The evaluation is done on a dataset from a project focused on the behaviour of students and how this impacts on their academic performance. We focus on those annotations concerning locations and movements of students, and we evaluate the annotations quality by checking their consistency. Results show that students are highly consistent with respect to the random baseline, and that these results can be improved by exploiting the semantics of annotations.

Labeling of Activity Recognition Datasets: Detection of Misbehaving Users

“…The consistency of annotations related to data collected by sensors on a smartphone was studied in [10]. The main goal was to relate the daily behavior of students with their academic performance, using information about their locations and movements.…”

Fixing Mislabeling by Human Annotators Leveraging Conflict Resolution and Prior Knowledge

“…Such an assumption is becoming less and less sustainable in a world where non-expert users are involved in annotating a constant stream of data, be it the flow of news in the Web (see, e.g., [10]), or the flow of sensor data in pervasive and ubiquitous computing (see, e.g., [15,17]). In the case of the latter scenario, the same input can be proposed multiple times, resulting in it being often labeled differently even by the same user [18].…”

“…Such an assumption is becoming less and less sustainable in a world where non-expert users are involved in annotating a constant stream of data, be it the flow of news in the Web (see, e.g., [10]), or the flow of sensor data in pervasive and ubiquitous computing (see, e.g., [15,17]). In the case of the latter scenario, the same input can be proposed multiple times, resulting in it being often labeled differently even by the same user [18].The research presented here is part of, and motivated by, a long-term series of experiments (two experiments per year for two years) aimed at studying the University student life and at correlating the students' behaviors with their academic performance, measured concerning grades and credits passed. The long-term goal is to support students who should increase their performance, thus also minimizing the possibility of a failure in getting the degree.…”