Evaluating Recommender Systems for Technology Enhanced Learning: A Quantitative Survey

Abstract

The increasing number of publications on recommender systems for Technology Enhanced Learning (TEL) evidence a growing interest in their development and deployment. In order to support learning, recommender systems for TEL need to consider specific requirements, which differ from the requirements for recommender systems in other domains like e-commerce. Consequently, these particular requirements motivate the incorporation of specific goals and methods in the evaluation process for TEL recommender systems. In this article, the diverse evaluation methods that have been applied to evaluate TEL recommender systems are investigated. A total of 235 articles are selected from major conferences, workshops, journals and books where relevant work have been published between 2000 and 2014. These articles are quantitatively analysed and classified according to the following criteria: type of evaluation methodology, subject of evaluation, and effects measured by the evaluation. Results from the survey suggest that there is a growing awareness in the research community of the necessity for more elaborate evaluations. At the same time, there is still substantial potential for further improvements. This survey highlights trends and discusses strengths and shortcomings of the evaluation of TEL recommender systems thus far, thereby aiming to stimulate researchers to contemplate novel evaluation approaches.

User-Centric Evaluation of Recommender Systems in Social Learning Platforms: Accuracy is Just the Tip of the Iceberg

“…Martin [7] claimed in his keynote to the ACM RecSys 2009 conference that around 50% of a recommender's commercial success goes to the aspects of "Human-Recommender Interaction" while the algorithm matters for 5% only (Martin2009). The importance of the user perspective has been realized even more in the educational domain [1], [8], [9]. Indeed, the main goal of the educational recommender systems extends well beyond accurate predictions and should also take into account quality metrics such as usefulness, novelty, or diversity of the recommendations.…”

“…Moreover, many user-centric evaluations are conducted using crowdsourcing. Although that is a valid approach, it has its limitations [9], [12]. In crowdsourcing, tasks, reliability and accuracy of the collected feedback data is sometimes questionable since there are of course differences between "cheap labor" workers and expensive experts [13].…”

Αξιοποίηση Συστημάτων Συστάσεων Στο Μαθησιακό Σχεδιασμό

“…osrportal.eu/en/repository). Ο τύπος της αξιολόγησης που πραγματοποιήθηκε σύμφωνα με την κατάταξη των Erdt, Fernandez, & Rensing (2015) είναι αξιολόγηση εκτός σύνδεσης (offline experiment). Οι (Fazeli, Drachsler, Brouns, & Sloep, 2014) προτείνουν ένα ΣΣ για την υποστήριξη των εκπαιδευτικών στη διαμοίραση ΜΑ με την εξατομικευμένη σύσταση ΜΑ σύμφωνα με το δίκτυο εμπιστοσύνης (trust networks) που διαμορφώνεται γύρω από αυτούς.…”

Intelligent e-Learning Systems: An Educational Paradigm Shift

“…In 2015, Erdt and others [14] proposed a quantitative survey on evaluating recommender systems. Recommender systems Technology Enhanced Learning need specific requirements which are different from that of e-commerce.…”