Simulated Annealing Model Search for Subset Selection in Screening Experiments

Abstract

The analysis of screening experiments based on nonregular designs can lead to a model selection problem in which the number of variables is large, the number of trials is small, and there are constraints on model structure. Common subset selection methods do not perform well in this setting. We propose a new approach particularly well suited to screening. The method uses an intentionally nonconvergent stochastic search to generate a large set of well-fitting models, each with the same number of variables. Model selection is then viewed as a feature extraction problem from this set. An easy-to-use graphical method and an automatic approach are proposed to determine the best models. Computer code and additional supplementary materials are available online.

Screening Strategies in the Presence of Interactions

“…For any individual simulated dataset, we would expect an expert user to outperform our analyses, for example, by using the graphical methods of Wolters and Bingham (2011) to better interpret the results from SAMS.…”

“…Wolters and Bingham (2011) suggested an entropy criterion for selecting a single model but found that, in some cases, multiple models may have very similar entropy values. In this article, we select a final set of active effects by comparing the threshold t to the average absolute estimated coefficient for each effect; here, the average is computed across all models visited by the SA algorithm that were found to include the effect.…”

“…The threshold, throughout, was set as t = μ act − 3.5σ act , so that it was linked to the active effect distribution on each run of the simulation. For SAMS, we used the recommended settings from Wolters and Bingham (2011); in particular, we set oversized models to contain n/3 + 3 effects. For the Bayesian analysis of supersaturated designs and stage 2 group screening, the values of τ and c, given in Table 1, were chosen for each of the four settings of active and inactive effects using the graphical methods of Section 3.2 to compromise between sensitivity and Type I error rate.…”

The choice of screening design

“…It may be argued that in our way of analysis, we did not take into account some of the attractive properties of the MinResIV designs and the DSDs, especially that main effects can be estimated unbiased from 2‐factor interactions and for the DSDs also unbiased from quadratic effects. Also, it is well demonstrated, for instance Wolters and Bingham, that the “correct model” is not necessarily the one with the lowest MSE, but that it is normally among the ones with the lowest MSE. Hence, our reported success frequencies may, and especially for the MinResIV 12 and the DSD 12 designs, be a little pessimistic.…”

Design of Experiments for Screening

“…Other methods for variable selection that have been effective for these designs include Bayesian methods that use mixture prior distributions for the elements of β [41,26] and the application of stochastic optimisation algorithms [110].…”