Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current crowd working platforms, and more accurate than in the traditional model.

Quality of Experience (QoE) typically involves conducting experiments in which stimuli are presented to participants and their judgments as well as behavioral data are collected. Nowadays, many experiments require software for the presentation of stimuli and the data collection from participants. While different software solutions exist, these are not tailored to conduct experiments on QoE. Moreover, replicating experiments or repeating the same experiment in different settings (e. g., laboratory vs. crowdsourcing) can further increase the software complexity.TheFragebogen is an open-source, versatile, extendable software framework for the implementation of questionnairesespecially for research on QoE. Implemented questionnaires can be presented with a state-of-the-art web browser to support a broad range of devices while the use of a web server being optional. Out-of-the-box, TheFragebogen provides graphical exact scales as well as free-hand input, the ability to collect behavioral data, and playback multimedia content.

No abstract

No abstract

No abstract