SciencOmat: A Gamified Research Platform for Evaluating Visual Attractiveness

Zagel, Christian; Piazza, Alexander; Petrov, Yoan; Bodendorf, Freimut

doi:10.1007/978-3-319-60486-2_5

Cited by 3 publications

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2) preference aggregation (Volkovs and Zemel 2012), e.g., online image-rating, 3 which aggregates ratings for the attractiveness of photos provided by different users (Knight and Keith 2005;Zagel et al 2018); and (3) meta-search (Desarkar et al 2016), which combines the search results of multiple search engines.…”

Section: Related Workmentioning

confidence: 99%

Stagewise learning for noisy k-ary preferences

2018

View full text Add to dashboard Cite

The aggregation of k-ary preferences is a novel ranking problem that plays an important role in several aspects of daily life, such as ordinal peer grading, online imagerating, meta-search and online product recommendation. Meanwhile, crowdsourcing is increasingly emerging as a way to provide a plethora of k-ary preferences for these types of ranking problems, due to the convenience of the platforms and the lower costs. However, preferences from crowd workers are often noisy, which inevitably degenerates the reliability of conventional aggregation models. In addition, traditional inferences usually lead to massive computational costs, which limits the scalability of aggregation models. To address both of these challenges, we propose a reliable CrowdsOUrced Plackett-LucE (COUPLE) model combined with an efficient Bayesian learning technique. To ensure reliability, we introduce an uncertainty vector for each crowd worker in COUPLE, which recovers the ground truth of the noisy preferences with a certain probability. Furthermore, we propose an Online Generalized Bayesian Moment Matching (OnlineGBMM) algorithm, which ensures that COUPLE is scalable to large-scale datasets. Comprehensive experiments on four large-scale synthetic datasets and three real-world datasets show that, COUPLE with OnlineGBMM achieves substantial improvements in reliability and noisy worker detection over other well-known approaches.

show abstract