Experimentation and Start-up Performance: Evidence from A/B Testing

Abstract

Recent scholarship argues that experimentation should be the organizing principle for entrepreneurial strategy. Experimentation leads to organizational learning, which drives improvements in firm performance. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of testing business ideas. Our results provide the first evidence on how digital experimentation affects a large sample of high-technology start-ups using data that tracks their growth, technology use, and products. We find that, although relatively few firms adopt A/B testing, among those that do, performance improves by 30%–100% after a year of use. We then argue that this substantial effect and relatively low adoption rate arises because start-ups do not only test one-off incremental changes, but also use A/B testing as part of a broader strategy of experimentation. Qualitative insights and additional quantitative analyses show that experimentation improves organizational learning, which helps start-ups develop more new products, identify and scale promising ideas, and fail faster when they receive negative signals. These findings inform the literatures on entrepreneurial strategy, organizational learning, and data-driven decision making. This paper was accepted by Toby Stuart, entrepreneurship and innovation.

The Modern Digital Operating Model

“…Many did not use it intensively. 2 If experiments work, where does experimentation go wrong? For one, meaningless metrics lead to pointless experiments.…”

Optimal Design of Controlled Experiments for Personalized Decision Making in the Presence of Observational Covariates

“…In fact, applying * Corresponding author. controlled experiments such as A/B testing experiments in business has improved the performance of companies [18]. Also, there are some methodological contributions for optimal design of experiments applied for A/B testing.…”

The Experimental Hand: How Platform-based Experimentation Reconfigures Worker Autonomy

“…Over the past decade, however, the organizational management of experimentation has dramatically changed (Luca & Bazerman, 2021;Van de Poel, Asveld, & Mehos, 2017). Scholars have shown that organizations can now leverage advances in digital technologies (e.g., algorithms, big data analytics, and machine learning) to deploy experiments at scale, in real-time, and at a relatively low cost (Koning, Hasan, & Chatterji, 2022;Marres & Stark, 2020;Zuboff, 2019). This new organizational management of experimentation is particularly salient in digital platform settings.…”

“…Understanding the impact of experimentation in digital platform settings on worker autonomy is important because most management research focuses on the design and efficacy of experimentation from an organization's vantage point (Engels, Wentland, & Pfotenhauer, 2019;Koning et al, 2022;Luca & Bazerman, 2021). Research has largely overlooked how organizations' use of experimentation impacts less powerful workers (i.e., the experimental subjects) beyond an experiment's pre-defined goal (Marres & Stark, 2020).…”

“…In addition, our study introduces an alternative conceptualization of worker autonomy in platform settings in which worker autonomy is subject to unexpected but accretive changes over time (e.g., via experimentation), regardless of a workers' status in an organization or field-level position. We also contribute to the emerging stream of research on the organizational management of experimentation (Kellogg, 2022;Koning et al, 2022;Marres & Stark, 2020).…”