Error Rates and Powers in Genome-Scale RNAi Screens

Abstract

For hit selection in genome-scale RNAi research, we do not want to miss small interfering RNAs (siRNAs) with large effects; meanwhile, we do not want to include siRNAs with small or no effects in the list of selected hits. There is a strong need to control both the false-negative rate (FNR), in which the siRNAs with large effects are not selected as hits, and the restricted false-positive rate (RFPR), in which the siRNAs with no or small effects are selected as hits. An error control method based on strictly standardized mean difference (SSMD) has been proposed to maintain a flexible and balanced control of FNR and RFPR. In this article, the authors illustrate how to maintain a balanced control of both FNR and RFPR using the plot of error rate versus SSMD as well as how to keep high powers using the plot of power versus SSMD in RNAi high-throughput screening experiments. There are relationships among FNR, RFPR, Type I and II errors, and power. Understanding the differences and links among these concepts is essential for people to use statistical terminology correctly and effectively for data analysis in genome-scale RNAi screens. Here the authors explore these differences and links. (Journal of Biomolecular Screening 2009:230-238) Key words: restricted false-positive rate, false-negative rate, strictly standardized mean difference, Type I error, Type II error, power INTRODUCTION R NAI IS A MECHANISM THAT KNOCKS DOWN GENES with complementary nucleotide sequences of doublestranded RNA. [1][2][3] In genome-scale RNAi screen experiments, the primary interest is (1) the assessment of the magnitude of impact on a biological response related to the knockdown of a gene and (2) the selection of siRNAs with large effects on the biological response of interest. The key is to search an analytic metric to effectively quantify knockdown effect and then to construct a selection criterion based on this metric to control false-positive and false-negative rates. Strictly standardized mean difference (SSMD) has been proposed and adopted for both quality control [4][5][6] and assessment of siRNA effects in RNAi high-throughput screening (HTS) assays. [7][8][9][10] Two clear advantages of using SSMD to asses siRNA effects are that (1) SSMD has both an original and probability meaning, and (2) its value is comparable across experiments. 4,8,10 Based on SSMD, an error control method has been proposed to maintain a flexible and balanced control of the false-negative rate (FNR), in which the siRNAs with large effects are not selected as hits, and the restricted false-positive rate (RFPR), in which the siRNAs with small or no effects are selected as hits. 8 In this article, we illustrate how to maintain a balanced control of both FNR and RFPR using the plot of error rate versus SSMD cutoff (or critical value) 10 as well as how to keep high powers using the plot of power versus SSMD critical value in RNAi HTS experiments. FNR and RFPR may be linked to Type I and II error rates under certain conditions, and they are also related to stat...

Statistical methods for analysis of high-throughput RNA interference screens

“…The Strictly Standardized Mean Difference metric discussed earlier can be employed for hit identification by screeners concerned with controlling the rate at which siRNAs that have real large or moderate effects fail to be identified as screening positives as well as the rate at which siRNAs that should be considered negative are identified as screening positives32. Formulae are provided33,34 for calculating the SSMD limits for hit selection based on the desired false positive level, false negative level, or both; while these require a large number of negative controls (> 50), follow-up work31 provided suggested SSMD cut-offs for screens without large numbers of negative samples, such as confirmatory screens.…”

The virus as metabolic engineer

“…This data has only begun to be interpreted, with recent reviews comparing the results of the HIV [40–44] and influenza screens [45]. In addition, these screens have emphasized the importance of experimental design and analysis in RNAi-based screens [46–49]. …”

Transcriptomic analysis of native versus cultured human and mouse dorsal root ganglia focused on pharmacological targets

“…SSMD is the difference of means controlled by the variance of the sample measurements. We used SSMD as a secondary effect size since it is well suited for small sample sizes as in our human samples [39; 71], while simultaneously taking into account the dispersion of the data points. For determining SSMD thresholds that identify genes that are systematically changing between conditions, we use the notion of the related Bhattacharyya coefficient [6], which is used to calculate the amount of overlap in the area under the curve of the two sample distributions in order to control for false positives in differential expression analysis.…”