Network A/B Testing

Gui, Huan; Xu, Ya; Bhasin, Anmol; Han, Jiawei

doi:10.1145/2736277.2741081

Cited by 98 publications

(28 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Under a fully general arbitrary interference model, it has been repeatedly shown that it is impossible to estimate any desired causal estimands as the model is not fully identifiable ( 3 , 19 – 21 ). As a result, there have been many proposed models that impose assumptions on exposure functions ( 3 , 19 , 22 – 24 ), interference neighborhoods ( 4 , 5 , 25 , 26 ), parametric structure ( 6 , 8 , 18 , 27 , 28 ), or a combination of these. Each of these assumptions leads to a different solution concept.…”

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

“…Unfortunately, this approach does not apply when the network could be highly connected, limiting its use in practice. The bias of standard estimators will scale with the number of edges across clusters, leading to proposed cluster randomized designs that randomize over clusters that are constructed to minimize edges between clusters ( 6 , 28 ).…”

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

“…While the above models impose network-based conditions on the interference, an alternate approach is to impose parametric structure on the form of the potential outcomes. The most common assumption is that the potential outcomes are linear with respect to a specified statistic of the local neighborhood ( 8 , 18 , 27 , 28 , 37 , 38 ). For example, ref.…”

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

“…For example, ref. 28 assumes that network effects arise when at least θ neighbors are treated, where θ is assumed to be known,

…”

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

See 3 more Smart Citations

Estimating the total treatment effect in randomized experiments with unknown network structure

Airoldi

Borgs

et al. 2022

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Randomized experiments are widely used to estimate the causal effects of a proposed treatment in many areas of science, from medicine and healthcare to the physical and biological sciences, from the social sciences to engineering, and from public policy to the technology industry. Here we consider situations where classical methods for estimating the total treatment effect on a target population are considerably biased due to confounding network effects, i.e., the fact that the treatment of an individual may impact its neighbors’ outcomes, an issue referred to as network interference or as nonindividualized treatment response. A key challenge in these situations is that the network is often unknown and difficult or costly to measure. We assume a potential outcomes model with heterogeneous additive network effects, encompassing a broad class of network interference sources, including spillover, peer effects, and contagion. First, we characterize the limitations in estimating the total treatment effect without knowledge of the network that drives interference. By contrast, we subsequently develop a simple estimator and efficient randomized design that outputs an unbiased estimate with low variance in situations where one is given access to average historical baseline measurements prior to the experiment. Our solution does not require knowledge of the underlying network structure, and it comes with statistical guarantees for a broad class of models. Due to their ease of interpretation and implementation, and their theoretical guarantees, we believe our results will have significant impact on the design of randomized experiments.

show abstract

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

“…For example, ref. 28 assumes that network effects arise when at least θ neighbors are treated, where θ is assumed to be known,

…”

Section: Alternate Approaches In the Literaturementioning

confidence: 99%

See 2 more Smart Citations

Estimating the total treatment effect in randomized experiments with unknown network structure

Airoldi

Borgs

et al. 2022

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…These tested models serve as a solution approaching an answer, (4) closing the gap in understanding for many questions in novel ways. For example, data-driven science has developed novel, large-scale methodologies relevant across scientific domains, including experimental design [8,74,102] and randomization techniques [56,99]. Figure 2 highlights several levels of modeling that can impact the final capability for answering a question of interest.…”

Section: Data Science Motivations For Network Structure Inferencementioning

confidence: 99%

Network Structure Inference, A Survey

2018

View full text Add to dashboard Cite

Networks represent relationships between entities in many complex systems, spanning from online social interactions to biological cell development and brain connectivity. In many cases, relationships between entities are unambiguously known: are two users "friends" in a social network? Do two researchers collaborate on a published article? Do two road segments in a transportation system intersect? These are directly observable in the system in question. In most cases, relationships between nodes are not directly observable and must be inferred: Does one gene regulate the expression of another? Do two animals who physically co-locate have a social bond? Who infected whom in a disease outbreak in a population? Existing approaches for inferring networks from data are found across many application domains and use specialized knowledge to infer and measure the quality of inferred network for a specific task or hypothesis. However, current research lacks a rigorous methodology that employs standard statistical validation on inferred models. In this survey, we examine (1) how network representations are constructed from underlying data, (2) the variety of questions and tasks on these representations over several domains, and (3) validation strategies for measuring the inferred network's capability of answering questions on the system of interest.

show abstract

The HURRIER process for experimentation in business‐to‐business mission‐critical systems

Mattos

Dakkak

Bosch

et al. 2021

J Software Evolu Process

View full text Add to dashboard Cite

Continuous experimentation (CE) refers to a set of practices used by software companies to rapidly assess the usage, value, and performance of deployed software using data collected from customers and systems in the field using an experimental methodology. However, despite its increasing popularity in developing web‐facing applications, CE has not been studied in the development process of business‐to‐business (B2B) mission‐critical systems. By observing the CE practices of different teams, with a case study methodology inside Ericsson, we were able to identify the different practices and techniques used in B2B mission‐critical systems and a description and classification of the four possible types of experiments. We present and analyze each of the four types of experiments with examples in the context of the mission‐critical long‐term evolution (4G) product. These examples show the general experimentation process followed by the teams and the use of the different CE practices and techniques. Based on these examples and the empirical data, we derived the HURRIER process to deliver high‐quality solutions that the customers value. Finally, we discuss the challenges, opportunities, and lessons learned from applying CE and the HURRIER process in B2B mission‐critical systems.

show abstract

Network A/B Testing

Cited by 98 publications

References 26 publications

Estimating the total treatment effect in randomized experiments with unknown network structure

Estimating the total treatment effect in randomized experiments with unknown network structure

Network Structure Inference, A Survey

The HURRIER process for experimentation in business‐to‐business mission‐critical systems

Contact Info

Product

Resources

About