Continuous Experimentation Scenarios: A Case Study in e-Commerce

Ros, Rasmus; Bjarnason, Elizabeth

doi:10.1109/seaa.2018.00064

Cited by 5 publications

(12 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A prior case study by Ros and Bjarnason (2018) outlines how Apptus uses continuous experimentation to improve their product in several scenarios: validating that a change has the intended outcome, manual optimization, and algorithmic optimization. Thus, Apptus is experienced with using optimization in various forms to optimize the web shops.…”

Section: Validation Company and E-commercementioning

confidence: 99%

“…First, a systematic mapping study (Ros and Runeson 2018) on continuous experimentation identified suitable algorithms to the problem domain and the assumptions in the optimization formulation as stated in Section 3. Second, an interview study with five participants from the validation company was conducted (Ros and Bjarnason 2018). The interviews investigated the difference between experimentation and optimization approaches, such as finding benefits and challenges of the respective approaches.…”

Section: Problem Identificationmentioning

confidence: 99%

“…This process is similar to one that is reportedly used for developing recommender systems at Netflix (Amatriain 2013) and Apptus (Ros and Bjarnason 2018). Simulations are also used at both companies to evaluate changes to their recommender system in fast feedback cycles.…”

Section: From Continuous Experimentation To Continuous Optimizationmentioning

confidence: 99%

See 2 more Smart Citations

Data-driven software design with Constraint Oriented Multi-variate Bandit Optimization (COMBO)

Ros

Hammar²

2020

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Context Software design in e-commerce can be improved with user data through controlled experiments (i.e. A/B tests) to better meet user needs. Machine learning-based algorithmic optimization techniques extends the approach to large number of variables to personalize software to different user needs. So far the optimization techniques has only been applied to optimize software of low complexity, such as colors and wordings of text. Objective In this paper, we introduce the COMBO toolkit with capability to model optimization variables and their relationship constraints specified through an embedded domain-specific language. The toolkit generates personalized software configurations for users as they arrive in the system, and the configurations improve over time in in relation to some given metric. COMBO has several implementations of machine learning algorithms and constraint solvers to optimize the model with user data by software developers without deep optimization knowledge. Method The toolkit was validated in a proof-of-concept by implementing two features that are relevant to Apptus, an e-commerce company that develops algorithms for web shops. The algorithmic performance was evaluated in simulations with realistic historic user data. Results The validation shows that the toolkit approach can model and improve relatively complex features with many types of variables and constraints, without causing noticeable delays for users. Conclusions We show that modeling software hierarchies in a formal model facilitates algorithmic optimization of more complex software. In this way, using COMBO, developers can make data-driven and personalized software products.

show abstract

Section: Validation Company and E-commercementioning

confidence: 99%

Section: Problem Identificationmentioning

confidence: 99%

See 1 more Smart Citation

Data-driven software design with Constraint Oriented Multi-variate Bandit Optimization (COMBO)

Ros

Hammar²

2020

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

show abstract

“…Cycles of such experiments can lead to constant improvements in product quality and usability [5]. Continuous experimentation has been studied to see whether it is possible to apply in various challenging different software segments [15], such as for automotives where there are technical obstacles to overcome [8], or businessto-business companies where there are organizational obstacles [14] and possibly a lack of access to user data [23].…”

Section: Introductionmentioning

confidence: 99%

“…Companies in many different sectors have adopted continuous experimentation [6,15] where features are evaluated with user feedback. Prototypes of an implementation can be quickly validated with users before implementation.…”

Section: Introductionmentioning

confidence: 99%

Continuous Experimentation with Product-Led Business Models: A Comparative Case Study

Ros

2021

Lecture Notes in Business Information Processing

Self Cite

View full text Add to dashboard Cite

Context. Continuous experimentation is used by many companies to improve their products with users data. In this study, the efficacy of continuous experimentation as used in two different types of business models (product-led or sales-led ). Method. Two case companies with a product-led business model and three companies with sales-led business model were compared against each other. 14 interviewees were selected from the cases. Results. Having a product-led business model enabled four different drivers to continuous experimentation: 1) development and sales & marketing integration, 2) improved prioritization, 3) decreased feature bloat, and 4) product measurability. Conclusions. The takeaway message is that a company must be structured in the right way to obtain benefits from experimentation.

show abstract

A theory of factors affecting continuous experimentation (FACE)

Ros,

Bjarnason,

Runeson

2023

Empir Software Eng

View full text Add to dashboard Cite

Context Continuous experimentation (CE) is used by many companies with internet-facing products to improve their business models and software solutions based on user data. Some companies deliberately adopt a systematic experiment-driven approach to software development while some companies use CE in a more ad-hoc fashion. Objective The goal of this study is to identify factors for success in CE that explain the variations in the utility and efficacy of CE between different companies. Method We conducted a multi-case study of 12 companies involved with CE and performed 27 interviews with practitioners at these companies. Based on that empirical data, we then built a theory of factors at play in CE. Results We introduce a theory of Factors Affecting Continuous Experimentation (FACE). The theory includes three factors, namely 1) processes and infrastructure for CE, 2) the user problem complexity of the product offering, and 3) incentive structures for CE. The theory explains how these factors affect the effectiveness of CE and its ability to achieve problem-solution and product-market fit. Conclusions Our theory may inspire practitioners to assess an organisation’s potential for adopting CE and to identify factors that pose challenges in gaining value from CE practices. Our results also provide a basis for defining practitioner guidelines and a starting point for further research on how contextual factors affect CE and how these may be mitigated.

show abstract

Continuous Experimentation Scenarios: A Case Study in e-Commerce

Cited by 5 publications

References 10 publications

Data-driven software design with Constraint Oriented Multi-variate Bandit Optimization (COMBO)

Data-driven software design with Constraint Oriented Multi-variate Bandit Optimization (COMBO)

Continuous Experimentation with Product-Led Business Models: A Comparative Case Study

A theory of factors affecting continuous experimentation (FACE)

Contact Info

Product

Resources

About