Skeptical Learning—An Algorithm and a Platform for Dealing with Mislabeling in Personal Context Recognition

Abstract

Mobile Crowd Sensing (MCS) is a novel IoT paradigm where sensor data, as collected by the user’s mobile devices, are integrated with user-generated content, e.g., annotations, self-reports, or images. While providing many advantages, the human involvement also brings big challenges, where the most critical is possibly the poor quality of human-provided content, most often due to the inaccurate input from non-expert users. In this paper, we propose Skeptical Learning, an interactive machine learning algorithm where the machine checks the quality of the user feedback and tries to fix it when a problem arises. In this context, the user feedback consists of answers to machine generated questions, at times defined by the machine. The main idea is to integrate three core elements, which are (i) sensor data , (ii) user answers, and (iii) existing prior knowledge of the world, and to enable a second round of validation with the user any time these three types of information jointly generate an inconsistency. The proposed solution is evaluated in a project focusing on a university student life scenario. The main goal of the project is to recognize the locations and transportation modes of the students. The results highlight an unexpectedly high pervasiveness of user mistakes in the university students life project. The results also shows the advantages provided by Skeptical Learning in dealing with the mislabeling issues in an interactive way and improving the prediction performance.

Lifelong Personal Context Recognition

“…Skeptical learning [18] is a recent interactive learning strategy that tackles annotation inconsistencies. The machine asks the user to revise her annotations if it is confident that there is an inconsistency [18,19]. The machine's uncertainty can be estimated using Bayesian [20] or frequentist [16] techniques, while enabling the AI to explain its skepticism by showing past examples that support the model's suspicion [21].…”

“…(iv) The assumption of using labels rather than full text is quite limiting. Some simple versions of this problem have been dealt by the Skeptical Learning [19] and are based on the use of a large multilingual resource, called UKC [26], but we are just at the beginning. The above is at the level of language.…”