The problem of evaluating automated large-scale evidence aggregators

Abstract

In the biomedical context, policy makers face a large amount of potentially discordant evidence from different sources. This prompts the question of how this evidence should be aggregated in the interests of best-informed policy recommendations. The starting point of our discussion is Hunter and Williams' recent work on an automated aggregation method for medical evidence. Our negative claim is that it is far from clear what the relevant criteria for evaluating an evidence aggregator of this sort are. What is the appropriate balance between explicitly coded algorithms and implicit reasoning involved, for instance, in the packaging of input evidence? In short: What is the optimal degree of 'automation'? On the positive side: We propose the ability to perform an adequate robustness analysis (which depends on the nature of the input variables and parameters of the aggregator) as the focal criterion, primarily because it directs efforts to what is most important, namely, the structure of the algorithm and the appropriate extent of automation. Moreover, where there are resource constraints on the aggregation process, one must also consider what balance between volume of evidence and accuracy in the treatment of individual evidence best facilitates inference. There is no prerogative to aggregate the total evidence available if this would in fact reduce overall accuracy. B Nicolas Wüthrich

Evidence amalgamation in the sciences: an introduction

“…Wüthrich and Steele (2018) are also interested in defending the utility of meta-analytic methods, especially in cases where automation becomes necessary due to the super-large-scale data involved. In these cases, careful problem solving cannot be done on a case-by-case basis and must instead be built into the automated aggregation algorithm; they suggest amenability to robustness analysis (Sect.…”

“…Roughly speaking, such algorithms probabilistically explore how one's conclusions change for different input and parameter values. Wüthrich and Steele (2018) argue that evidence amalgamation algorithms ought to be assessed by considering the kind of robustness analysis, thusly understood, that can be performed; the possibility space associated with the robustness analysis is revealing of the basic structure of the algorithm.…”

“…Wüthrich and Steele (2018) are concerned with evaluating procedures for aggregating ever-larger bodies of evidence. The rapid growth of available data will eventually require the application of automated evidence aggregators implemented as computer algorithms.…”

“…So, by "robustness analysis" Wüthrich and Steele (2018) refer to a relatively simple notion of robustness in terms of varying parameter values or "dial settings." They argue that designing large-scale evidence aggregation algorithms with the possibility of such robustness analysis has two crucial advantages: it becomes clear from early on in the design process (i) which (types of) uncertainty can be subjected to robustness analysis by "turning the dial" and (ii) which types of uncertainty cannot be subjected to robustness analysis because there is no "dial" one could turn.…”

“…Thus, it also increases transparency of the inner workings of the algorithm. Wüthrich and Steele (2018) point out that in actual practice, further complications arise due to limited (computational and other data processing) resources. An evidence aggregation algorithm may produce better results by a deeper analysis of a smaller data set than by a shallower analysis of a larger data set.…”