Registered Report: A Laboratory Experiment on Using Different Financial-Incentivization Schemes in Software-Engineering Experimentation

Abstract

Empirical studies in software engineering are often conducted with open-source developers or in industrial collaborations. Seemingly, this resulted in few experiments using financial incentives (e.g., money, vouchers) as a strategy to motivate the participants' behavior; which is typically done in other research communities, such as economics or psychology. Even the current version of the SIGSOFT Empirical Standards does mention payouts for completing surveys only, but not for mimicking the real-world or motivating realistic behavior during experiments. So, there is a lack of understanding regarding whether financial incentives can or cannot be useful for software-engineering experimentation. To tackle this problem, we plan a survey based on which we will conduct a controlled laboratory experiment. Precisely, we will use the survey to elicit incentivization schemes we will employ as (up to) four payoff functions (i.e., mappings of choices or performance in an experiment to a monetary payment) during a code-review task in the experiment: (1) a scheme that employees prefer, (2) a scheme that is actually employed, (3) a scheme that is performance-independent, and (4) a scheme that mimics an open-source scenario. Using a between-subject design, we aim to explore how the different schemes impact the participants' performance. Our contributions help understand the impact of financial incentives on developers in experiments as well as real-world scenarios, guiding researchers in designing experiments and organizations in compensating developers.